fundamentals

Documentation Status Coverage Status

Some project setup tools including logging, settings and database connections.

Here’s a summary of what’s included in the python package:

Classes

fundamentals.files.fileChunker The fileChunker iterator - iterate over large line-based files to reduce memory footprint
fundamentals.logs.GroupWriteRotatingFileHandler rotating file handler for logging
fundamentals.logs.emptyLogger A fake logger object so user can set ``log=False`` if required
fundamentals.mysql.database a database object that can setup up a ssh tunnel (optional) and a database connection
fundamentals.mysql.sqlite2mysql Take a sqlite database file and copy the tables within it to a MySQL database
fundamentals.mysql.yaml_to_database Take key-values from yaml files including a tablename(s) and add them to a mysql database table
fundamentals.renderer.list_of_dictionaries The dataset object is a list of python dictionaries.Using this class, the data can be rendered as various list and markup formats.
fundamentals.tools common setup methods & attributes of the main function in cl-util
fundamentals.utKit Default setup for fundamentals style unit-testing workflow (all tests base on nose module)

Functions

fundamentals.download._dump_files_to_local_drive takes the files stored in memory and dumps them to the local drive
fundamentals.download._fetch Retrieve an HTML document or file from the web at a given URL
fundamentals.download.append_now_datestamp_to_filename append the current datestamp to the end of the filename (before the extension).
fundamentals.download.extract_filename_from_url get the filename from a URL.
fundamentals.download.get_now_datetime_filestamp A datetime stamp to be appended to the end of filenames: ``YYYYMMDDtHHMMSS``
fundamentals.download.multiobject_download get multiple url documents and place them in specified download directory/directories
fundamentals.files.recursive_directory_listing list directory contents recursively.
fundamentals.fmultiprocess multiprocess pool
fundamentals.logs.console_logger Setup and return a console logger
fundamentals.logs.setup_dryx_logging setup dryx style python logging
fundamentals.mysql.convert_dictionary_to_mysql_table convert dictionary to mysql table
fundamentals.mysql.directory_script_runner A function to run all of the mysql scripts in a given directory (in a modified date order, oldest first) and then act on the script file in accordance with the succcess or failure of its execution
fundamentals.mysql.get_database_table_column_names get database table column names
fundamentals.mysql.insert_list_of_dictionaries_into_database_tables insert list of dictionaries into database tables
fundamentals.mysql.readquery Given a mysql query, read the data from the database and return the results as a list of dictionaries (database rows)
fundamentals.mysql.setup_database_connection Start a database connection using settings in yaml file
fundamentals.mysql.table_exists Probe a database to determine if a given table exists
fundamentals.mysql.writequery Execute a MySQL write command given a sql query
fundamentals.times.calculate_time_difference Return the time difference between two dates as a string
fundamentals.times.get_now_sql_datetime A datetime stamp in MySQL format: ``YYYY-MM-DDTHH:MM:SS``

Multimarkdown Rendering Test Document

Table of Contents

Code and Syntax Highlighting

Inline code has back-ticks around it.

var s = "JavaScript syntax highlighting";
alert(s);
s = "Python syntax highlighting"
print s
myString = """Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."""
No language indicated, so no syntax highlighting. 
But let's throw in a <b>tag</b>.

Mermaid

gantt
        dateFormat  YYYY-MM-DD
        title Adding GANTT diagram functionality to mermaid
        section A section
        Completed task            :done,    des1, 2014-01-06,2014-01-08
        Active task               :active,  des2, 2014-01-09, 3d
        Future task               :         des3, after des2, 5d
        Future task2               :         des4, after des3, 5d
        section Critical tasks
        Completed task in the critical line :crit, done, 2014-01-06,24h
        Implement parser and jison          :crit, done, after des1, 2d
        Create tests for parser             :crit, active, 3d
        Future task in critical line        :crit, 5d
        Create tests for renderer           :2d
        Add to mermaid                      :1d

Flowchart

s=>start: start
e=>end: end
o=>operation: operation
sr=>subroutine: subroutine
c=>condition: condition
i=>inputoutput: inputoutput
p=>parallel: parallel

s->o->c
c(yes)->i->e
c(no)->p
p(path1, bottom)->sr(right)->o
p(path2, top)->o

Tables

Colons can be used to align columns.

Tables Are Cool
col 3 is right-aligned $1600
col 2 is centered $12
zebra stripes are neat $1

Definitions

term
definition

Transclusion

If MMD transclusion is working you will see a unicorn here:

{{multimarkdown_transclusion_test_20190121111324.md}}

If iA transclusion is working you will see a unicorn here:

Linking to an Adjacent File

Here is another file in the same directory

Math

A formula, \({e}^{i\pi }+1=0\), inside a paragraph.

\[{e}^{i\pi }+1=0\]

Super/Sub Scripts

m2

x2,y

xz

C6H12O6

Citations

Cite a source.[p. 42, 1]

Black (2015)[2]

CriticMarkup

This {++is ++}a test.

This is {–is –}a test.

This {~~isn’t>is} a test.

This is a {==test==}.

This is a test.{>>What is it a test of?<<}

Paragraphs

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.

Cross References

Intro

Some text

Conclusion

Link to Intro.

Link to the end.

Task Lists

  • Completed task item
  • Unfinished task item

Footnote

Here’s a sentence with a footnote3 in the middle of it!

Text.4

Headers

# H1
## H2
### H3
#### H4
##### H5
###### H6

Emphasis

Emphasis, aka italics, with asterisks or underscores.
Strong emphasis, aka bold, with asterisks or underscores.
Strikethrough uses two tildes. Scratch this.

Lists

  1. First ordered list item

  2. Another item
    • Unordered sub-list.
  3. Actual numbers don’t matter, just that it’s a number
    1. Ordered sub-list
  4. And another item.

    You can have properly indented paragraphs within list items.

and unordered lists:

  • Unordered list can use asterisks
  • Or minuses
  • Or pluses

I’m an inline-style link

I’m an inline-style link with title

I’m a reference-style link

I’m a relative reference to a repository file

You can use numbers for reference-style link definitions

Or leave it empty and use the link text itself.

URLs and URLs in angle brackets will automatically get turned into links. http://www.example.com or http://www.example.com and sometimes example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

Images

Here’s our logo (hover to see the title text):

Inline-style: alt text

Reference-style:

Blockquotes

Blockquotes are very handy in email to emulate reply text. This line is part of the same quote.

Quote break.

This is a very long line that will still be quoted properly when it wraps. Oh boy let’s keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can put Markdown into a blockquote.

Inline HTML

You can also use raw HTML in your Markdown, and it’ll mostly work pretty well.

Definition list
Is something people use sometimes.
Markdown in HTML
Does not work very well. Use HTML tags.

Horizontal Rule

Three or more…


Hyphens


  1. John Doe. A Totally Fake Book. Vanity Press, 2006.

  2. A Bryden Black 2015, The Lion, the Dove, & the Lamb, Wipf and Stock Publishers

  3. This is the footnote. ↩︎

  4. This is an inline footnote. ↩︎

Installation

The easiest way to install fundamentals is to use pip:

pip install fundamentals

Or you can clone the github repo and install from a local version of the code:

git clone git@github.com:thespacedoctor/fundamentals.git
cd fundamentals
python setup.py install

To upgrade to the latest version of fundamentals use the command:

pip install fundamentals --upgrade

Development

If you want to tinker with the code, then install in development mode. This means you can modify the code from your cloned repo:

git clone git@github.com:thespacedoctor/fundamentals.git
cd fundamentals
python setup.py develop

Pull requests are welcomed!

Issues

Please report any issues here.

Documentation

Documentation for fundamentals is hosted by Read the Docs (last stable version and latest version).

Tutorial

The most useful class of the fundamentals package is fundamentals.tools. The tools object, when added to the __main__ function of your cl-utils module, will do a lot of heavy lifting to set you up with some useful project tools. These include a logger, a default settings file and abilities to setup database connections and ssh tunnels.

For example, I can have a cl-utils.py module living at the root of my python package with the following content:

#!/usr/local/bin/python
# encoding: utf-8
"""
*CL utils for myProject*

Usage:
    myProjectCommand [-s <pathToSettingsFile>]

    -h, --help            show this help message
    -s, --settings        the settings file
"""
################# GLOBAL IMPORTS ####################
import sys
import os
os.environ['TERM'] = 'vt100'
import glob
from docopt import docopt
from fundamentals import tools, times


def main(arguments=None):
    """
    *The main function used when ``cl_utils.py`` is run as a single script from the cl, or when installed as a cl command*
    """
    # SETUP THE COMMAND-LINE UTIL SETTINGS
    su = tools(
        arguments=arguments,
        docString=__doc__,
        logLevel="DEBUG",
        options_first=False,
        projectName="myProject"
    )
    arguments, settings, log, dbConn = su.setup()

    # UNPACK REMAINING CL ARGUMENTS USING `EXEC` TO SETUP THE VARIABLE NAMES
    # AUTOMATICALLY
    for arg, val in arguments.iteritems():
        if arg[0] == "-":
            varname = arg.replace("-", "") + "Flag"
        else:
            varname = arg.replace("<", "").replace(">", "")
        if isinstance(val, str) or isinstance(val, unicode):
            exec(varname + " = '%s'" % (val,))
        else:
            exec(varname + " = %s" % (val,))
        if arg == "--dbConn":
            dbConn = val
        log.debug('%s = %s' % (varname, val,))

    ## START LOGGING ##
    startTime = times.get_now_sql_datetime()
    log.info(
        '--- STARTING TO RUN THE cl_utils.py AT %s' %
        (startTime,))

    #########
    # DO STUFF HERE!
    #########

    if "dbConn" in locals() and dbConn:
        dbConn.commit()
        dbConn.close()
    ## FINISH LOGGING ##
    endTime = times.get_now_sql_datetime()
    runningTime = times.calculate_time_difference(startTime, endTime)
    log.info('-- FINISHED ATTEMPT TO RUN THE cl_utils.py AT %s (RUNTIME: %s) --' %
             (endTime, runningTime, ))

    return

if __name__ == '__main__':
    main()

And within setup.py I will include:

entry_points={
    'console_scripts': ['myProjectCommand=myProject.cl_utils:main'],
}

And finally I also have a default-settings.yaml file at the root of my python package with the content:

version: 1
database settings:
    db: dryx_unit_testing
    host: localhost
    user: unittesting
    password: utpass
    sshPort: False
    loginPath: unittesting

# SSH TUNNEL - if a tunnel is required to connect to the database(s) then add setup here
# Note only one tunnel is setup - may need to change this to 2 tunnels in the future if
# code, static catalogue database and transient database are all on seperate machines.
ssh tunnel:
    use tunnel: True   # True | False
    remote user: username
    remote ip: mydomain.co.uk
    remote datbase host: mydatabaseName
    port: 9002

logging settings:
    formatters:
        file_style:
            format: '* %(asctime)s - %(name)s - %(levelname)s (%(pathname)s > %(funcName)s > %(lineno)d) - %(message)s  '
            datefmt: '%Y/%m/%d %H:%M:%S'
        console_style:
            format: '* %(asctime)s - %(levelname)s: %(pathname)s:%(funcName)s:%(lineno)d > %(message)s'
            datefmt: '%H:%M:%S'
        html_style:
            format: '<div id="row" class="%(levelname)s"><span class="date">%(asctime)s</span>   <span class="label">file:</span><span class="filename">%(filename)s</span>   <span class="label">method:</span><span class="funcName">%(funcName)s</span>   <span class="label">line#:</span><span class="lineno">%(lineno)d</span> <span class="pathname">%(pathname)s</span>  <div class="right"><span class="message">%(message)s</span><span class="levelname">%(levelname)s</span></div></div>'
            datefmt: '%Y-%m-%d <span class= "time">%H:%M <span class= "seconds">%Ss</span></span>'
    handlers:
        console:
            class: logging.StreamHandler
            level: DEBUG
            formatter: console_style
            stream: ext://sys.stdout
        file:
            class: logging.handlers.GroupWriteRotatingFileHandler
            level: WARNING
            formatter: file_style
            filename: /Users/Dave/.config/myProject/myProject.log
            mode: w+
            maxBytes: 102400
            backupCount: 1
    root:
        level: WARNING
        handlers: [file,console]

Once the package is installed, from the terminal I can now run:

myProjectCommand

The first time this command is run (no arguments), a dedicated project directory is setup at ~/.config/myProject/. The default-settings.yaml file is copied to ~/.config/myProject/myProject.yaml and a log file created at ~/.config/myProject/myProject.log. Subsequent running of this command will read the settings from ~/.config/myProject/myProject.yaml, so any changes made to this file will be seen by the command. It’s also possible to pass in alternate settings by running the command with the -s flag:

myProjectCommand -s /path/to/different-settings.yaml

I now have the following objects I can pass to the rest of my package:

  • settings – a dictionary of all the settings read from the yaml config file
  • log – a logger setup with the settings found in the yaml config file
  • dbConn – a database connect setup with the settings found in the yaml config file

Note

There are other usful trinkets within the code - read the docs to find out how to use them.

Command-Line Tools

mysqlSucker

Given a directory of MySQL scripts, mysqlSucker can execute the script files and process them according to their success or failure status.

Before you begin you will need to run the following code once to set a login-path for your mysql server:

mysql_config_editor set --login-path=<uniqueLoginName> --host=localhost --user=<myUsername> --password

This stores your credentials in an encrypted file located at ‘~/.mylogin.cnf’. Use mysql_config_editor print --all to see all of the login-paths set.

Usage

Usage:
    mysqlSucker <pathToDirectory> <loginPath> <databaseName> [-s successRule -f failureRule]

    pathToDirectory       path to the directory containing the sql scripts to run (scripts must have `.sql` extension)
    loginPath             the local-path as set with `mysql_config_editor` (`mysqlSucker -h` for more details)
    databaseName          the name of the database to execute the scripts within

Options:
    -h, --help                                  show this help message
    -s successRule, --success successRule       what to do if script succeeds. Default *None* [None|delete|subFolderName]
    -f failureRule, --failure failureRule       what to do if script fails. Default *None* [None|delete|subFolderName]

Examples

To simply execute the scripts in a directory run:

mysqlSucker /path/to/scriptdir myLoginPath myDatabaseName

To delete script after thay have executed successfully:

mysqlSucker /path/to/scriptdir myLoginPath myDatabaseName -s delete

To move successful script to a passed sub-directory of /path/to/scriptdir and failed scripts to a failed sub-directory

mysqlSucker /path/to/scriptdir myLoginPath myDatabaseName -s pass -f failed

yaml2db

You may be thinking why would you ever need to dump the contents of a yaml file into a database. Well … IFTTT! With the IFTTT service you have quick and super easy access to the APIs of dozens of your favourite web-services. You can use any of the IFTTT channels as an input for an applet that then writes a yaml file with the data you wish to capture to a directory in your Dropbox account. Pointing yaml2db at this Dropbox directory in your filesystem then sucks all this lovely data into a MySQL database. It’s up to you then what you want to do with the data, the point is you now have a simple, homogenised way of pouring content from all those APIs into a local database.

You could grab the URLs for videos you ‘like’ on youtube or add to your watch-later list on vimeo. Or how about your sleep log from your fitbit, or a new contact added to your iOS device.

How about this applet that logs URLs moved into my instapaper read-work-projects folder:

https://i.imgur.com/oDSgBhQ.png

So how does it work? yaml2db will take either a path to a single yaml file or to an entire directory of yaml files, parse the contents of those files and add the key-values to a mysql database. Here’s the usages statement (printed by typing yaml2db -h from a terminal window):

Usage:
    yaml2db [-d] -s <pathToSettingsFile> <pathToYaml>
    yaml2db [-d] --host=<host> --user=<user> --passwd=<passwd> --dbName=<dbName> <pathToYaml>

Options:

    pathToYaml            path to a single yaml file or directory of yaml files
    pathToSettingsFile    path to a settings file with logging and database information (yaml file)
    --host=<host>         the database host
    --user=<user>         database user
    --passwd=<passwd>     database user password
    --dbName=<dbName>     name of the database to add the table content to

    -d, --delete          delete yaml file(s) once added to datbase
    -h, --help            show this help message
    -s, --settings        path to a settings file with logging and database parameters

The yaml files require a table value to indicate the name of the database table (or tables; comma-separated) to add the yaml key-values to. If the database tables do not exist, yaml2db will create them for you. Here’s an example of the content of a yaml file:

title: Why you should do most of your text editing in : Sublime Text | Sublime Text Tips
url: http://sublimetexttips.com/why-you-should-do-most-of-your-text-editing-in-sublime-text/?utm_source=drip&utm_medium=email&utm_campaign=editor-proliferation
kind: webpage
subtype: article
table: web_articles,podcasts

yaml2db will take the content of this file and add it to both a web_articles and podcasts table.

https://i.imgur.com/sfPUfNj.png

To execute yaml2db on a single yaml file run:

yaml2db -d --host=localhost --user=username --passwd=mypassword --dbName=myGreatDatabase /Users/Me/yaml/20161219105124.yaml

Obviously replace the database parameters with your own. The -d flag indicates that you want to delete the yaml file once it’s been parsed.

To run yaml2db on an entire directory of yaml files run:

yaml2db -s /Users/Me/mydefault_settings.yaml /Users/Me/yaml/

where /Users/Me/mydefault_settings.yaml contains my database and logging parameters (see example at the start of this tutorial).

sqlite2mysql

The sqlite2mysql tool does exactly what you would expect it to do; it takes the path to a sqlite database file and copies the tables found in the database to a MySQL database of your choosing.

For the usage run sqlite2mysql -h from the command-line:

Take a sqlite database file and copy the tables within it to a MySQL database

Usage:
    sqlite2mysql -s <pathToSettingsFile> <pathToSqliteDB> [<tablePrefix>]

Options:

    pathToSqliteDB        path to the sqlite database file
    tablePrefix           a string to prefix the table names when copying to mysql database
    pathToSettingsFile    path to a settings file with logging and database information (yaml file)

    -h, --help            show this help message
    -v, --version         show version
    -s, --settings        the settings file

As usual pathToSettingsFile contains your MySQL database settings (see example at the start of this tutorial). To convert a sqlite databse I could run something like:

sqlite2mysql -s /Users/Me/mydefault_settings.yaml /Users/Me/ebooks.sqlite imported

This command takes the tables in the /Users/Me/ebooks.sqlite database, prepends the names of the tables with ‘imported_’ and imports the tables into the MySQL database whose settings are found in the /Users/Me/mydefault_settings.yaml file. Here are the resulting MySQL database tables as listed in MySQLWorkbench:

imported tables in MySQL database

Python Package Documentation