Python, pre-commit, and black

If you've had trouble configuring a python project with local repo copies of black and get errors you can't explain, this may help. Maybe.

I ran into some config issues somewhere between my pre-commit config and my black config. In case someone else runs into this I wanted to leave some bread crumbs.

The issue. Black keeps exiting with "exit code: 123" after it tries formatting non-python files. It then told me it couldn't format files that weren't python files: yaml, md, and toml. I'd thought I'd excluded everything I needed to, but something was still tripping me up.

BLUF (bottom line up front). Specify "types: [python]" for black to make it parse only python files.

The setup. This is a small personal python project with pandas, numpy, and xlsxwriter. Because in my day job my test environment is completely disconnected from the Internet I wanted to set this up with "repo: local" so I could test that kind of config. There's just not a ton of info I've found online on using local pre-commit hooks, which is funny in its own way. I'm not saying that there's /no/ documentation, since https://pre-commit.com/#repository-local-hooks exists and Anthony Sottile of pre-commit fame has commented on many stackoverflow.com questions. I just had trouble finding "the answer."

I'm on a System76 laptop running Pop!_OS 22.04 (hopefully they'll finish the COSMIC desktop soon!), so it defaults to python 3.10-vintage software.

The errors. My pre-commit config parsed fine, as did the pyproject.toml file for black.

(.venv) andrew@bun-bun:~/src/myproj$ pre-commit validate-config
(.venv) andrew@bun-bun:~/src/myproj$ pre-commit

black....................................................................Failed

- hook id: black
- exit code: 123

error: cannot format .pre-commit-config.yaml: Cannot parse for target version Python 3.10: 1:6: repos:

error: cannot format README.md: Cannot parse for target version Python 3.10: 4:15: andrew@bun-bun: [text from this line]

error: cannot format pyproject.toml: Cannot parse for target version Python 3.10: 20:8:   [text from this other line]

Oh no! 💥 💔 💥
3 files failed to reformat.

check-toml...............................................................Passed

Well, at least I got check-toml configured correctly!

The configs. I've not tried this setup before and I'm still unclear about what's supposed to be in pyproject.toml and what's supposed to be in .pre-commit-config.yaml so I'd not copy and paste this in toto. I started with .pre-commit-config.yaml:

repos:

-   repo: local
    hooks:

    -   id: black
        name: black
        entry: black
        language: python
        types: [python]
        args: [--line-length=95]
        exclude: .yaml$

    -   id: check-toml
        name: check-toml
        language: system
        entry: check-toml
        files: toml

The toml config seemed to work, at least after I did a "pip install pre-commit-hooks" at some point. But if you leave out the "types" line, black tries to do all files, which you don't want. My "exclude:" line also didn't work, but I see as I type this up that I had "exclude: .yaml^" instead of "exclude: .yaml$", so that's a bit of a mess. Whoops!

I tried to exclude various text files in my pyproject.toml file, too, but it was the "types: [python]" line that really did the trick. Here's my [tool.black] section. Some I've obviously grabbed from somewhere else (who needs to exclude both git and mercurial directories?) but I was willing to try. The data I'm munging (various excel spreadsheets) lives under "data" so I wanted to exclude all of that. (The data is also not checked into git.)

[tool.black]
line-length = 95
target-version = ['py310']
include = '\.pyi?$'
exclude = '''
    /(
        \.git
      | \.hg
      | \.mypy_cache
      | \.tox
      | \.venv
      | _build
      | buck-out
      | build
      | dist
      | \.toml
      | \.md
      | \.yaml
      | data
    )/
    '''

Final trick. If you're trying to figure out what labels the files have, use "identify-cli". Some file checks are "AND", some are "OR."

This'll give you some idea of what popular values are.

(.venv) andrew@bun-bun:~/src/myproj$ identify-cli .pre-commit-config.yaml
["file", "non-executable", "text", "yaml"]
(.venv) andrew@bun-bun:~/src/myproj$ identify-cli library.py
["file", "non-executable", "python", "text"]
(.venv) andrew@bun-bun:~/src/myproj$ identify-cli script.py
["executable", "file", "python", "text"]

CUPS and POP!_OS

Ok, sometimes I use this blog so I can find stuff later. So, Andrew, if you find you can't add a printer in System76's Pop!_OS because your login and password don't let you log into CUPS, try:

sudo usermod -aG lpadmin $USER

Courtesy of https://github.com/pop-os/pop/issues/1262.

Dual Booting with EFI, The Joy of

For several years I've purchased my laptops from System 76. They're good, and I'm on my third laptop. Many moons ago I needed a hardware swap-out while I was on a 2-week R&R from an Iraq deployment, and they hooked me right up. (I also live 30 minutes from where their HQ was at the time.) I learned the hard way that I needed Windows to control the booting of Windows and Linux (usually SUSE), or Windows patches would never work quite right.

EFI is great. I'd not realized that you dual-boot with the EFI bootloader, not with EasyBCD, boot.ini files, or the other stuff I'd used over the years. I did have to delete some fonts to clear out a little space in the /boot/efi partition, but worse things happen at sea, you know. I'm really pleased how easy it was. Hit ESC when booting and one-time boot your OS. Beautiful.

Lethality: The Podcast

I recorded two podcasts while at the U.S. Army War College. The second was on Lethality. In DoD terms, lethality means "good." It's the new hotness--all the rage. As an artilleryman, I approve.

But there's another side to these things. I've also studied a lot of economics, and it may just be easier to lose the peace as to lose the war. In that vein, I offer There's More to Life than Lethality. (Though the Space Force #pewpew article was more fun.)

Podcasting at the U.S. Army War College

I was able to be on two podcasts while a student at the U.S. Army War College (USAWC) over the last year. The first was aimed at a civilian academic audience. Basically, "What in the world is it like to be a military grad student at a 'War College'?" Yes, there are such things, and the things are a bit different in a professional military education (PME) setting than at a regular university. For example, the military is a direct consumer of its own product, so there is a very tight feedback loop, analyzing the output.

It was actually Part II of a two part series. Part I was an interview of Prof. Jacqueline Whitt, a professor at USAWC. She's also taught at the U.S. Air War College and at West Point. Those two interviews were conducted by Prof. Robert Farley of the University of Kentucky, who was a visiting professor last year at Carlisle Barracks.

The second one was way nerdier than the first, and a ton of fun. Who wants to talk Space Force? #pewpew The discussions between the Space Force, Space Corps, U.S. Space Command, and A.F. Space Command get really confusing, so Prof. Whitt and I talked about that in "What should a U.S. Space Force Look Like?" The short version: It's too soon, just have the Air Force work it out for now.

Python argparse with defaults -- in-script and importable

I've tried using argparse (and optparse before that) and have had trouble finding a workflow that made me happy. This one is pretty close. It allows you to make defaults for when you call a script on the command line and makes those same defaults available if you import the module. It works like this:

1. Make a dictionary with the defaults you want.
2. In the function you write that parses the arguments, use the dictionary values as the defaults.
3. Merge the defaults dictionary and the arguments dictionary together.
4. Return the merged dictionary.

The Gist version is available here, or you can see it below, too.

 #######################################

#!/usr/bin/python3
'''
> python3 args_example.py -h
usage: args_example.py [-h] [-f FIRST_OPTION] [-s SECOND_OPTION]
                       mandatory_argument

Example argument parser with default values.

positional arguments:
  mandatory_argument    Not everything can have a default value. This is a
                        mandatory argument.

optional arguments:
  -h, --help            show this help message and exit
  -f FIRST_OPTION, --first_option FIRST_OPTION
                        This is the first option. Default:
                        first_default_value.
  -s SECOND_OPTION, --second_option SECOND_OPTION
                        This is the second option. (Default:
                        second_default_value)
  -t THIRD_OPTION, --third_option THIRD_OPTION
                        This is the third option. (Default:
                        third_default_value)

> python3 args_example.py -f "non-default first value" --second_option 'non-default second value' "this is mandatory"                                                      
{'first_option': 'non-default first value',
 'mandatory_argument': 'this is mandatory',
 'second_option': 'non-default second value',
 'third_option': 'third_default_value'}

Of course, https://pymotw.com/2/argparse/ is still excellent.
'''


import argparse
import pprint


default_settings = {'first_option': 'first_default_value',
                    'second_option': 'second_default_value',
                    'third_option': 'third_default_value'}


def get_args(settings=default_settings):
    '''Reads from sys.stdin. Use stdin to get new options, and return options
    merged with the default options. Settings is a dictionary. We'll have an
    argument (no, you came here for an argument), set two options, and default
    one.
    '''

    parser = argparse.ArgumentParser(description='Example argument parser with default values.')
    parser.add_argument('mandatory_argument', help='Not everything can have a default value. This is a mandatory argument.')
    parser.add_argument('-f', '--first_option', help='This is the first option. Default: %(default)s.',
                        default=settings['first_option'])
    parser.add_argument('-s', '--second_option', help='This is the second option. (Default: %(default)s)',
                        default=settings['second_option'])
    parser.add_argument('-t', '--third_option', help='This is the third option. (Default: %(default)s)',
                        default=settings['third_option'])
    args = parser.parse_args()
    args_settings = vars(args) # everything is easier with a dictionary
 
    #python3.5+
    merged_settings_3 = {**settings, **args_settings}

    #python2
    merged_settings_2 = default_settings.copy()
    merged_settings_2.update(args_settings)

    merged_settings_brute_force = '''
    if args.first_option:
        settings['first_option'] = args.first_option
    if args.second_option:
        settings['second_option'] = args.second_option
    if args.third_option:
        settings['third_option'] = args.third_option
    settings['mandatory_argument'] = args.mandatory_argument
    return merged_settings_brute_force
        '''

    return merged_settings_3


def main():
    settings = get_args()
    pprint.pprint(settings)


if __name__ == '__main__':
    main()

Cat facts & python

I'd not actually used anything with json before, but (unsurprisingly) the requests library came through.

>>> import requests
>>> url = 'http://catfacts-api.appspot.com/api/facts?number=1'
>>> print(requests.get(url).json()['facts'][0])
Cat families usually play best in even numbers. Cats and kittens should be acquired in pairs whenever possible.

While that gets you the answer, when did you ever really just want the answer? That's pretty uninteresting. Let's get there in stages.

This is straightforward (yes, I'm assuming you have requests installed already):

>>> import requests
>>> url = 'http://catfacts-api.appspot.com/api/facts?number=1'

Then we can get the cat fact:
 >>> cat_fact = requests.get(url)

Which looks like this:
>>> cat_fact.text
'{"facts": ["A cat\'s whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."], "success": "true"}'

Well, we really just want the string of the dictionary item in the string. The json() method makes it a dictionary, not a string.
 >>> cat_fact.json()
{'facts': ["A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."], 'success': 'true'}





And we can use the dictionary key ('facts')...
>>> print(cat_fact.json()['facts'])
["A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."]

But we don't want a list, we want the string. So take the first item in the list.
>>> print(cat_fact.json()['facts'][0])
A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through.

We could also iterate through a number of cat facts, including a spurious use of f-string formatting.
>>> number = 3
>>> url_number = f'http://catfacts-api.appspot.com/api/facts?number={number}'
>>> catfact = requests.get(url_number).json()
>>> catfact
{'facts': ['Not every cat gets "high" from catnip. Whether or not a cat responds to it depends upon a recessive gene: no gene, no joy.', 'Cats have an average of 24 whiskers, arranged in four horizontal rows on each side.', 'The strongest climber among the big cats, a leopard can carry prey twice its weight up a tree.'], 'success': 'true'}

>>> for fact in catfact['facts']:
...   print(fact)
...
Not every cat gets "high" from catnip. Whether or not a cat responds to it depends upon a recessive gene: no gene, no joy.
Cats have an average of 24 whiskers, arranged in four horizontal rows on each side.
The strongest climber among the big cats, a leopard can carry prey twice its weight up a tree.

Which I think is quite enough about cat facts.