Tuesday, June 27, 2017

Python argparse with defaults -- in-script and importable

I've tried using argparse (and optparse before that) and have had trouble finding a workflow that made me happy. This one is pretty close. It allows you to make defaults for when you call a script on the command line and makes those same defaults available if you import the module. It works like this:
  1. Make a dictionary with the defaults you want.
  2. In the function you write that parses the arguments, use the dictionary values as the defaults.
  3. Merge the defaults dictionary and the arguments dictionary together.
  4. Return the merged dictionary.
The Gist version is available here, or you can see it below, too.

 #######################################

#!/usr/bin/python3
'''
> python3 args_example.py -h
usage: args_example.py [-h] [-f FIRST_OPTION] [-s SECOND_OPTION]
                       mandatory_argument

Example argument parser with default values.

positional arguments:
  mandatory_argument    Not everything can have a default value. This is a
                        mandatory argument.

optional arguments:
  -h, --help            show this help message and exit
  -f FIRST_OPTION, --first_option FIRST_OPTION
                        This is the first option. Default:
                        first_default_value.
  -s SECOND_OPTION, --second_option SECOND_OPTION
                        This is the second option. (Default:
                        second_default_value)
  -t THIRD_OPTION, --third_option THIRD_OPTION
                        This is the third option. (Default:
                        third_default_value)

> python3 args_example.py -f "non-default first value" --second_option 'non-default second value' "this is mandatory"                                                      
{'first_option': 'non-default first value',
 'mandatory_argument': 'this is mandatory',
 'second_option': 'non-default second value',
 'third_option': 'third_default_value'}

Of course, https://pymotw.com/2/argparse/ is still excellent.
'''


import argparse
import pprint


default_settings = {'first_option': 'first_default_value',
                    'second_option': 'second_default_value',
                    'third_option': 'third_default_value'}


def get_args(settings=default_settings):
    '''Reads from sys.stdin. Use stdin to get new options, and return options
    merged with the default options. Settings is a dictionary. We'll have an
    argument (no, you came here for an argument), set two options, and default
    one.
    '''

    parser = argparse.ArgumentParser(description='Example argument parser with default values.')
    parser.add_argument('mandatory_argument', help='Not everything can have a default value. This is a mandatory argument.')
    parser.add_argument('-f', '--first_option', help='This is the first option. Default: %(default)s.',
                        default=settings['first_option'])
    parser.add_argument('-s', '--second_option', help='This is the second option. (Default: %(default)s)',
                        default=settings['second_option'])
    parser.add_argument('-t', '--third_option', help='This is the third option. (Default: %(default)s)',
                        default=settings['third_option'])
    args = parser.parse_args()
    args_settings = vars(args) # everything is easier with a dictionary
 
    #python3.5+
    merged_settings_3 = {**settings, **args_settings}

    #python2
    merged_settings_2 = default_settings.copy()
    merged_settings_2.update(args_settings)

    merged_settings_brute_force = '''
    if args.first_option:
        settings['first_option'] = args.first_option
    if args.second_option:
        settings['second_option'] = args.second_option
    if args.third_option:
        settings['third_option'] = args.third_option
    settings['mandatory_argument'] = args.mandatory_argument
    return merged_settings_brute_force
        '''

    return merged_settings_3


def main():
    settings = get_args()
    pprint.pprint(settings)


if __name__ == '__main__':
    main()

Saturday, June 10, 2017

Cat facts & python

I'd not actually used anything with json before, but (unsurprisingly) the requests library came through.

>>> import requests
>>> url = 'http://catfacts-api.appspot.com/api/facts?number=1'
>>> print(requests.get(url).json()['facts'][0])
Cat families usually play best in even numbers. Cats and kittens should be acquired in pairs whenever possible.

While that gets you the answer, when did you ever really just want the answer? That's pretty uninteresting. Let's get there in stages.

This is straightforward (yes, I'm assuming you have requests installed already):

>>> import requests
>>> url = 'http://catfacts-api.appspot.com/api/facts?number=1'

Then we can get the cat fact:
 >>> cat_fact = requests.get(url)

Which looks like this:
>>> cat_fact.text
'{"facts": ["A cat\'s whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."], "success": "true"}'

Well, we really just want the string of the dictionary item in the string. The json() method makes it a dictionary, not a string.
 >>> cat_fact.json()
{'facts': ["A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."], 'success': 'true'}





And we can use the dictionary key ('facts')...
>>> print(cat_fact.json()['facts'])
["A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through."]

But we don't want a list, we want the string. So take the first item in the list.
>>> print(cat_fact.json()['facts'][0])
A cat's whiskers are thought to be a kind of radar, which helps a cat gauge the space it intends to walk through.

We could also iterate through a number of cat facts, including a spurious use of f-string formatting.
>>> number = 3
>>> url_number = f'http://catfacts-api.appspot.com/api/facts?number={number}'
>>> catfact = requests.get(url_number).json()
>>> catfact
{'facts': ['Not every cat gets "high" from catnip. Whether or not a cat responds to it depends upon a recessive gene: no gene, no joy.', 'Cats have an average of 24 whiskers, arranged in four horizontal rows on each side.', 'The strongest climber among the big cats, a leopard can carry prey twice its weight up a tree.'], 'success': 'true'}

>>> for fact in catfact['facts']:
...   print(fact)
...
Not every cat gets "high" from catnip. Whether or not a cat responds to it depends upon a recessive gene: no gene, no joy.
Cats have an average of 24 whiskers, arranged in four horizontal rows on each side.
The strongest climber among the big cats, a leopard can carry prey twice its weight up a tree.

Which I think is quite enough about cat facts.

Friday, March 4, 2016

Free laws!

This is something I've been looking for for a long time. Colorado is now letting you have a complete copy of the law for free! http://coloradofoic.org/colorado-statutes-database-should-be-free-lawmakers-decide/ has more information about it. Years ago I'd made a fancy pdf of the Colorado Constitution with LaTeX, but this has the amazing benefit that I don't have to do it. You can download your very own Colorado Constitution from http://tornado.state.co.us/gov_dir/leg_dir/olls/2015titles.htm. Slightly strange: the constitution download is under the Colorado Revised Statutes. The link for the constitution is a web only copy.

Update: The link moved. https://leg.colorado.gov/agencies/office-legislative-legal-services/2015-crs-titles-download

Thursday, September 3, 2015

CSV reader examples for python 3

Among other things, I like python. I quite like the Talk Python to Me podcast, done by Michael Kennedy. He is, among other things, a python trainer. Among his online python training videos is one on parsing csv (comma separated value) files. His example didn't actually use the python csv module, so I whipped up a couple of examples that used csv.reader and csv.DictReader.

These use python 3.4.

=================== csv.reader example: #!c:\\anaconda3\\python.exe '''taken from https://www.youtube.com/watch?v=qajONCIvhEc https://support.spatialkey.com/spatialkey-sample-csv-data/ http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv ''' import csv import os def main(): print("Hello File I/O minicast using csv.reader") filename = os.path.abspath(os.path.join("data", "Sacramentorealestatetransactions.csv")) print(filename) with open(filename, 'r') as csvfile: fin = csv.reader(csvfile, delimiter=',') header = next(fin) print(header) # ['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude'] entries = [] for line in fin: # lines look like # ['3526 HIGH ST', 'SACRAMENTO', '95838', 'CA', '2', '1', '836', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '59222', '38.631913', '-121.434879'] row = dict() for i, h in enumerate(header): row[h] = line[i] entries.append(row) entries.sort(key= lambda r: -1*int(r['price'])) # Entries are a list of dicts. Each line looks like: # {'street': '9401 BARREL RACER CT', 'state': 'CA', 'sq__ft': '4400', 'city': 'WILTON', 'beds': '4', 'baths': '3', 'sale_date': 'Fri May 16 00:00:00 EDT 2008', 'price': '884790', 'longitude': '-121.194858', 'latitude': '38.415298', 'zip': '95693', 'type': 'Residential'} for e in entries[:5]: print("{0} beds, {1} baths sold for ${2:,}".format( e['beds'], e['baths'], int(e['price']) )) if _name_ == '__main__': main()

======================= csv.DictReader example:
#!c:\\anaconda3\\python.exe '''taken from https://www.youtube.com/watch?v=qajONCIvhEc https://support.spatialkey.com/spatialkey-sample-csv-data/ http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv ''' import csv import os def main(): print("Hello File I/O minicast using csv.DictReader") filename = os.path.abspath(os.path.join("data", "Sacramentorealestatetransactions.csv")) print(filename) with open(filename, 'r') as csvfile: fin = csv.DictReader(csvfile, delimiter=',') print(fin.fieldnames) # ['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude'] entries = list(fin) # lines/rows in fin look like # {'street': '3882 YELLOWSTONE LN', 'state': 'CA', 'sq__ft': '1362', 'city': 'EL DORADO HILLS', 'beds': '3', 'baths': '2', 'sale_date': 'Thu May 15 00:00:00 EDT 2008', 'price': '235738', 'longitude': '-121.075915', 'latitude': '38.655245', 'zip': '95762', 'type': 'Residential'} entries.sort(key= lambda r: -1*int(r['price'])) for e in entries[:5]: print("{0} beds, {1} baths sold for ${2:,}".format( e['beds'], e['baths'], int(e['price']) )) if _name_ == '__main__': main()

Thursday, March 26, 2015

Fixing Linux Firefox DEE (mail.mil)

It's a constant battle to be able to receive my Defense Enterprise Email on linux. (Heck, it's not that far off on Windows, either.) I had it working, then it stopped, and militarycac.com came to the rescue. It turns out that you need to renegotiate the SSL session while authenticating, which is a security hole. https://militarycac.com/firefox.htm had the right info for me:

You'll need to enable SSL renegotiation, do this by pointing your browser to about:config. After confirming that you know what you are doing, you need to start typing in:
security.ssl.allow_unrestricted_renego_everywhere__temporarily_available_pref
set it to true (by double clicking it). After this you should be able to access the site.
 I didn't even need to restart firefox! Now I can be in linux more often, and not just when I think I won't need to check email. DoD is moving to an all-in-one outlook web access (OWA) client, and it is much finickier than normal CAC (smart card) enabled web sites.

Monday, March 16, 2015

Renaming files in Windows

Sometimes the small things are the most helpful. I needed to rename a couple dozen jpg files to "...small.jpg". Through the joys of Internet searching, I pretty quickly came to a solution from http://www.howtogeek.com/111859/how-to-batch-rename-files-in-windows-4-ways-to-rename-multiple-files/ using Windows PowerShell. So with a quick:

PS > dir | Rename-Item -NewName {$_.name -replace ".jpg","Small.jpg"}

FirstPicture.jpg became FirstPictureSmall.jpg. I hadn't installed The Gimp yet on this laptop, but I found through a YouTube video (http://www.youtube.com/watch?v=19GbI6oFrW0) that you can use Right Click > Send To > Mail Recipient, and Outlook (assuming it's installed) will offer to change the size for you.

Saturday, June 14, 2014

Destinations vs Capabilities

Brian Enke has another good article, this time on how NASA is treating things around the Mars mission. I've never been a fan of the plan to go to an asteroid. Yes, it's technically difficult, but unless they're going to figure out how to mine it by going there, just go to Mars or the Moon. The latest twist on the Mars mission is to test out things in Lunar orbit.

What I liked about the article was the discussion on Capabilities vs Destinations, since for years I've been trained to talk about what capability is needed, rather than the solutions asked for. For example, don't say that you need five people to clean up an abandoned parking lot that's been used for a dump, say that you need the capability of moving x tons of trash in y amount of time, and repaving a parking lot of z size. While this is a little different from a "Destination" I wanted to see what he said.

And the answer is ... destinations are cheaper than capabilities. I think this is true because you can pool your trade space when you're looking at the destination, goal, or end state. When you list out a bunch of individual capabilities, the tendency is to over-engineer each one, they aren't exactly made to interface with each other, and you still need the destination at the end, because a rocket, habitat, and a life support system through space don't magically come together for the mission.