Introduction

What I'm talking about today?

  • Client, scope, and where we come in
  • Technology -> Jupyter Notebooks -> Python
  • Status & Future Goals

Who We are

What We're Doing?

  • Agency of Education

  • "Enterprise Environtment"

  • Jupyter & Python

Past & Current Status

Past & Current Status

  • Turnover and Leadership Vacuum
    • Head of AOE ADS "Re-assigned"
    • Half of ADS staff has left since we started
    • Many AOE staff have also left
    • Access and CSV(Excell) Everywhere
      • Processes have high touch points, are not cost effective, and error prone.
      • Silos of knowledge, documentation, and willpower

Past & Current Status

  • Everyone knows a little
    • SQL is everywhere and most staff can leverage it
    • A few have experience with R and other similar scripting languages
  • SLDS (State Longitudinal Database) Project caused headaches
    • 3 year project, finished last month after 6.5 years
    • Source of all data from the field (aka schools)
    • Other aditional data sources that require manual processes to load
  • Reporting - Who cares about what they do?
    • Federal -> Grants/Funding
    • Public/Legislative -> Budget & Legal requirements, Taxes...
    • Internal -> Ad-Hoc reporting needed to run day to day operations

How can we help?

The Statement of Work(SOW)

Support desired project and program outcomes through analysis, design, development, and implementation of major internal and external systems and sub-systems integration and automation work.

Serve as expert support for technical and data staff at project meetings and act as liaison between users and vendors

Collaborate with management to develop, document, and implement system strategies and standards. May act as server administrator. May act in a team leadership fashion for other agency development staff for duration of the work.

Takeaway: Broad Scope, Freedom to make descisions

Short Term

  • Get the SLDS under state resources -> Reduce Risk and increases access to data
  • Reduce risk on critical infrastrucutre through automation and auditing
  • Help people get their work done faster using automation -> Save Agency $$$

Long Term

  • Federal Generate Tool -> Reporting
  • Data Mart (CEDS) -> Compatibility
  • Power BI -> Reporting
  • Consistancy -> Sanity and Process Organization

Tools?

Needs

  • Scripts to Automate work
  • Flexible Licensing
  • Easy Developer Experience
  • Data Scientist Friendly
  • Broad availability of libraies and utilities
  • Fits in Micro$ofts ecosystem

Anyone have any ideas?

Jupyter

Documentation and Code in one place!

What is it?

  • Interactive Computing kind of like a REPL = Read–eval–print loop
  • At it's Core is a messaging protocal that allows communication with a "kernel"
  • Started with iPython, but has been extended to support multiple languages and GUI's
    • iCSharp, iJavascript, R
    • Lab, Nteract, PaperMill

Who else uses it?

  • Netflix, Micro$oft, IBM, Soundcloud, rackspace, etc...

Jupyter

Why it's a good fit

  • Automation - Data, software & browser
  • Opens up options to use the right tool for the job You can use what language makes sense
  • Connects to any DB system out there
  • Provides more options for input/output than SQL alone
  • Can be run on a schedule, and the output artifact is easy for users to review
  • Can produce interactive reports and GUI's

Demo

It's just JSON

\n

Future Status

  • CEDS - DataMart
    • Consistant way to access data people need to do their jobs
    • Specs to work with external parties going forward
  • Generate ETL process
    • Create a SSIS package to migrate data automatically for fedral reporting
    • Stand up Generate Tool on state resources
  • Remaking the world
    • Wrangle the code that determines our taxes! (Average Daily Membership)
    • Rewrite all internal process so they can be automated

Lessons Learned

  • Data Science and Python are so hot right now!
  • The state is slow
  • The ADS shake up a number of years ago is still having effects
  • People don't actually want to program
    • of a certain age
    • "not a programmer"
    • checked out emmotionally

When does it end?

Client responded with :)