Introduction
What I'm talking about today?
- Client, scope, and where we come in
- Technology -> Jupyter Notebooks -> Python
- Status & Future Goals
Who We are
- James Nadeau - Tech Lead - http://jamesnadeau.com
- Scott Lewis - Developer - needs to make his own website :)
What We're Doing?
-
Agency of Education
-
"Enterprise Environtment"
-
Jupyter & Python
Past & Current Status
Past & Current Status
- Turnover and Leadership Vacuum
- Head of AOE ADS "Re-assigned"
- Half of ADS staff has left since we started
- Many AOE staff have also left
- Access and CSV(Excell) Everywhere
- Processes have high touch points, are not cost effective, and error prone.
- Silos of knowledge, documentation, and willpower
Past & Current Status
- Everyone knows a little
- SQL is everywhere and most staff can leverage it
- A few have experience with R and other similar scripting languages
- SLDS (State Longitudinal Database) Project caused headaches
- 3 year project, finished last month after 6.5 years
- Source of all data from the field (aka schools)
- Other aditional data sources that require manual processes to load
- Reporting - Who cares about what they do?
- Federal -> Grants/Funding
- Public/Legislative -> Budget & Legal requirements, Taxes...
- Internal -> Ad-Hoc reporting needed to run day to day operations
How can we help?
The Statement of Work(SOW)
Support desired project and program outcomes through analysis, design, development, and implementation of major internal and external systems and sub-systems integration and automation work.
Serve as expert support for technical and data staff at project meetings and act as liaison between users and vendors
Collaborate with management to develop, document, and implement system strategies and standards. May act as server administrator. May act in a team leadership fashion for other agency development staff for duration of the work.
Takeaway: Broad Scope, Freedom to make descisions
Short Term
- Get the SLDS under state resources -> Reduce Risk and increases access to data
- Reduce risk on critical infrastrucutre through automation and auditing
- Help people get their work done faster using automation -> Save Agency $$$
Long Term
- Federal Generate Tool -> Reporting
- Data Mart (CEDS) -> Compatibility
- Power BI -> Reporting
- Consistancy -> Sanity and Process Organization
Tools?
Needs
- Scripts to Automate work
- Flexible Licensing
- Easy Developer Experience
- Data Scientist Friendly
- Broad availability of libraies and utilities
- Fits in Micro$ofts ecosystem
Anyone have any ideas?
Jupyter
Documentation and Code in one place!
What is it?
Interactive Computingkind of like a REPL = Read–eval–print loop- At it's Core is a messaging protocal that allows communication with a "kernel"
- Started with iPython, but has been extended to support multiple languages and GUI's
- iCSharp, iJavascript, R
- Lab, Nteract, PaperMill
Who else uses it?
- Netflix, Micro$oft, IBM, Soundcloud, rackspace, etc...
Jupyter
Why it's a good fit
- Automation - Data, software & browser
- Opens up options to use the right tool for the job You can use what language makes sense
- Connects to any DB system out there
- Provides more options for input/output than SQL alone
- Can be run on a schedule, and the output artifact is easy for users to review
- Can produce interactive reports and GUI's
Demo
It's just JSON
\n Future Status
- CEDS - DataMart
- Consistant way to access data people need to do their jobs
- Specs to work with external parties going forward
- Generate ETL process
- Create a SSIS package to migrate data automatically for fedral reporting
- Stand up Generate Tool on state resources
- Remaking the world
- Wrangle the code that determines our taxes! (Average Daily Membership)
- Rewrite all internal process so they can be automated
Lessons Learned
- Data Science and Python are so hot right now!
- The state is slow
- The ADS shake up a number of years ago is still having effects
- People don't actually want to program
- of a certain age
- "not a programmer"
- checked out emmotionally
When does it end?
Client responded with :)
