Data Visualization Toolkit Using JavaScript Rails and Postgres to Present Data and Geospatial Information Barrett Clark instant download
Data Visualization Toolkit Using JavaScript Rails and Postgres to Present Data and Geospatial Information Barrett Clark instant download
https://textbookfull.com/product/data-visualization-toolkit-
using-javascript-rails-and-postgres-to-present-data-and-
geospatial-information-barrett-clark/
https://textbookfull.com/product/source-code-analytics-with-
roslyn-and-javascript-data-visualization-mukherjee/
https://textbookfull.com/product/data-visualization-with-python-
and-javascript-scrape-clean-explore-transform-your-data-1st-
edition-kyran-dale/
https://textbookfull.com/product/earth-systems-data-processing-
and-visualization-using-matlab-zekai-sen/
https://textbookfull.com/product/practical-data-science-cookbook-
data-pre-processing-analysis-and-visualization-using-r-and-
python-prabhanjan-tattar/
Python Data Analysis: Perform data collection, data
processing, wrangling, visualization, and model
building using Python 3rd Edition Avinash Navlani
https://textbookfull.com/product/python-data-analysis-perform-
data-collection-data-processing-wrangling-visualization-and-
model-building-using-python-3rd-edition-avinash-navlani/
https://textbookfull.com/product/d3-js-in-action-data-
visualization-with-javascript-2nd-edition-elijah-meeks/
https://textbookfull.com/product/d3-js-in-action-data-
visualization-with-javascript-second-edition-elijah-meeks/
https://textbookfull.com/product/data-smart-using-data-science-
to-transform-information-into-insight-2nd-edition-jordan-
goldmeier/
https://textbookfull.com/product/data-visualization-representing-
information-on-modern-web-1st-edition-kirk/
About This E-Book
EPUB is an open, industry-standard format for e-books. However, support for EPUB and its many
features varies across reading devices and applications. Use your device or app settings to customize the
presentation to your liking. Settings that you can customize often include font, font size, single or double
column, landscape or portrait mode, and figures that you can click or tap to enlarge. For additional
information about the settings and features on your reading device or app, visit the device manufacturer’s
Web site.
Many titles include programming code or configuration examples. To optimize the presentation of these
elements, view the e-book in single-column, landscape mode and adjust the font size to the smallest
setting. In addition to presenting code and configurations in the reflowable text format, we have included
images of the code that mimic the presentation found in the print book; therefore, where the reflowable
format may compromise the presentation of the code listing, you will see a “Click here to view code
image” link. Click the link to view the print-fidelity code image. To return to the previous page viewed,
click the Back button on your device or app.
Data Visualization Toolkit
Using JavaScript, Rails™, and Postgres to Present Data and
Geospatial Information Barrett Clark
Boston • Columbus • Indianapolis • New York • San Francisco • Amsterdam • Cape Town
Dubai • London • Madrid • Milan • Munich • Paris • Montreal • Toronto • Delhi • Mexico City
São Paulo • Sydney • Hong Kong • Seoul • Singapore • Taipei • Tokyo
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and the publisher was aware of a trademark
claim, the designations have been printed with initial capital letters or in all capitals.
The author and publisher have taken care in the preparation of this book, but make no expressed or
implied warranty of any kind and assume no responsibility for errors or omissions. No liability is
assumed for incidental or consequential damages in connection with or arising out of the use of the
information or programs contained herein.
For information about buying this title in bulk quantities, or for special sales opportunities (which may
include electronic versions; custom cover designs; and content particular to your business, training goals,
marketing focus, or branding interests), please contact our corporate sales department at
[email protected] or (800) 382-3419.
For government sales inquiries, please contact [email protected].
For questions about sales outside the U.S., please contact [email protected].
Visit us on the Web: informit.com/aw
Library of Congress Control Number: 2016944665
Copyright © 2017 Pearson Education, Inc.
All rights reserved. Printed in the United States of America. This publication is protected by copyright,
and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a
retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying,
recording, or likewise. For information regarding permissions, request forms and the appropriate contacts
within the Pearson Education Global Rights & Permissions Department, please visit
www.pearsoned.com/permissions/.
ISBN-13: 978-0-13-446443-5
ISBN-10: 0-13-446443-5
Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana.
1 16
To my children. Never stop exploring and asking questions
(even when you wear me out).
Contents
Foreword
Preface
Acknowledgments
About the Author
Part I ActiveRecord and D3
Chapter 1 D3 and Rails
Your Toolbox—A Three-Ring Circus
Database
Application Server
Graphing Library
Maryland Residential Sales App
Evaluating Data
Data Fields
Simple Pie Chart
Summary
Chapter 2 Transforming Data with ActiveRecord and D3
Pie Chart Revisited
Legible Labels
Mouseover Effects
You Can Function
Bar Chart
New Views, New Routes
Bar Chart Controller Actions
Bar Chart JavaScript
Scatter Plot
Scatter Plot?
Scatter Plot Controller Actions
Scatter Plot Views and Routes
Scatter Plot JavaScript
Scatter Plot Revisited
Box Plot
Quartiles
Boxes, Whiskers, Circles, What?!
Box Plot Data and Views
Box Plot JavaScript
Summary
Chapter 3 Working with Time Series Data
Historic Daily Weather Data
Weather Rails App
Weather Readings Model
Weather Readings Import
Weather Stations Model
Weather Stations Import
Simple Line Graph
Weather Controller
Fetch the Data
The View Files
Draw the Line Graph
Tweak 1: Simple Multiline Graph
Tweak 2: Add Circle to Highlight the Maximum Temperature
Tweak 3: Add Circle to Highlight the Minimum Temperature
Tweak 4: Add Text to Display the Temperature Change
Tweak 5: Add a Line Between the Focus Circles
Summary
Chapter 4 Working with Large Datasets
Git and Large Files
The Cloud
Hotlinking
Benchmarking
Benchmark and Compare
Benchmark All the Things
Querying “Big Data”
Using Scopes in the Model
Adding Indices
When Benchmarks and Statistics Lie
Summary
Part II Using SQL in Rails
Chapter 5 Window Functions, Subqueries, and Common Table Expression
Why Use SQL?
Database Portability Is a Lie
Tripping Over ActiveRecord
User-Defined Functions
Why?
Heresy!
How?
How to Use SQL in Rails
Scatter Plot with Mortgage Payment
Window Functions
Window Functions Greatest Hits
Lead and Lag
Partitions
First Value and Last Value
Row Number
Using Subqueries
Common Table Expression
CTE and the Heatmap
The Query
The Controller and View
The JavaScript
Summary
Chapter 6 The Chord Diagram
The Matrix Is the Truth
Flight Departures Data
Departures App
Airports
Carriers
Departures
Transforming the Data
Fetching the Data
Generating the Matrix
Finalizing the Matrix
Create the Views
Departures Controller and Routes
Departures View
Departures Style
Draw the Chord Diagram
Disjointed City Pairs
Using the Lead Window Function to Find Empty Leg Flights
Optimizing Slow Queries with the Materialized View
Draw the Disjointed City Pairs Chord Diagram
Summary
Chapter 7 Time Series Aggregates in Postgres
Finding Flight Segments
Creating a Series of Time
Turning Data into Time Series Data
Graphing the Timeline
Basic Timeline
Fancy Timeline
Summary
Chapter 8 Using a Separate Reporting Database
Transactional versus Reporting Databases
Worker Processes
Postgres Schemas
Working with Multiple Schemas in Rails
Defining the Schema Connection
Creating a New Schema
Creating Objects in the Reporting Schema
Materialized View in the Reporting Schema
Tables in the Reporting Schema
Summary
Part III Geospatial Rails
Chapter 9 Working with Geospatial Data in Rails
GIS Primer
It’s (Longitude, Latitude) Not (Latitude, Longitude)
Decimal Degrees
Degrees, Minutes, Seconds (DMS)
Datum
Map Projection
Spatial Reference System Identifier (SRID)
Three Feature Types
PostGIS
Postgres Contrib Modules
Installing PostGIS
PostGIS Functions
ActiveRecord and PostGIS
ActiveRecord PostGIS Adapter
Rails PostGIS Configuration
PostGIS Hosting Considerations
Using Geospatial Data in Rails
Creating Geospatial Table Fields
Latitude and Longitude
Simple GIS Calculation
Working with Shapefiles
Shapefile Import Schema
Importing from a Shapefile
Shapefile ETL
Update Missing lonlat Data
Summary
Chapter 10 Making Maps with Leaflet and Rails
Leaflet
Map Tiles
Map Layers
Incorporating Leaflet into Rails to Visualize Weather Stations
Using a Separate Rails Layout for the Map
Map Controller
Map Index
Map Data GeoJSON View
Mapping the Weather Stations
Visualizing Airports
Markers
Marker Cluster
Drawing Flight Paths
Visualizing Zip Codes
Updating the Maryland Residential Sales App for PostGIS
Zip Code Geographies
Importing the Zip Code Shapefile
Mapping Zip Codes
Choropleth
Summary
Chapter 11 Querying Geospatial Data
Finding Items within a Bounding Box
What Is a Bounding Box?
Writing a Bounding Box Query
Writing a Bounding Box Query Using SQL and PostGIS
Writing a Bounding Box Query Using ActiveRecord
Finding Items within the Bounding Box
Finding Items Near a Point
Writing the Query
Using ActiveRecord
Calculating Distance
Summary
Afterword
Appendix A Ruby and Rails Setup
Install Ruby
Create the App
More Gems
Config Files
Finalize the Setup
Appendix B Brief Postgres Overview
Installing Postgres
From Source
Package Manager
Postgres.app
SQL Tools
Command Line
GUI Tool
Bulk Importing Data
COPY SQL Statement
\copy PSQL Command
pg_restore
The Query Plan
Appendix C SQL Join Overview
Join Example Database Setup
Inner Join
Left Outer Join
Right Outer Join
Full Outer Join
Cross Join
Self Join
Index
Foreword
I pitched Addison-Wesley on the idea of a Professional Ruby Series way back in 2005. As research for
this foreword, I dug up the original proposal and looked at the list of titles that I envisioned would make
up the series in the future. Wow, what an exercise. Out of a dozen, only one of those original ideas became
reality, the fantastic Design Patterns in Ruby book by my old friend Russ Olsen. Literally none of the
others have seen the light of day, including Extending Rails into the Enterprise, Behavior Driven
Development in Ruby, Software Testing with Ruby, and AJAX on Rails.
Okay, I admit that some of those ideas kind of sucked. However, one of them definitely did not suck and
I’ve been holding out for it since the beginning: Processing and Displaying Data in Ruby. The reason is
that as series editor, a big part of my job is to make sure that we publish books that stand the test of time.
That’s no small feat given the accelerating pace of change in technology. But I absolutely know that the
need to collect, transform, and intelligently display data is an eternal problem in computing. I was
positive that if we published an awesome book covering that topic, it would fill a vital need in the
marketplace and sell many copies year after year.
That need was still apparent a few years later when I led a team wrangling terabytes of credit card
transaction records using Ruby domain-specific languages at Barclays Bank. It was there for many of my
clients at Hashrocket, and it was there in every one of my subsequent startups.
The fact is, our world is being systematically flooded by data. Never mind the normal domain datasets
for most of the apps we write, it’s event data and time-series logging that is really exploding. Not only
that, but the looming IoT (Internet of Things) revolution will dramatically increase the amount of
information we need to deal with, probably by orders of magnitude. Which means more and more of us
will be asked to participate in making sense of that data by transforming and visualizing it in a way that
makes sense for stakeholders.
In other words, I’ve been waiting over ten years for this book and can barely wait any longer! Luckily,
Barrett Clark has made that wait worthwhile. He’s got over ten years of experience with Ruby on Rails,
and the depth of his knowledge shines through in his writing, which I’m glad to report is clear, concise,
and confident. There are also three (count ‘em) sample applications from which to draw examples—I’m
sure that readers who are newer to programming will appreciate the abundance of working code as
starting points for their own projects.
This isn’t the biggest book in the series, but it covers a lot of ground. I was actually a little worried that
it might cover too much ground when I first saw the outline. But it works, and as I was able to make my
way through the manuscript I realized why. Barrett has been practicing all of this stuff in his day job for
many years—Postgres, D3, GIS, all of it! The knowledge in this book is not just pulled together from
reference material and blog posts, it’s real-world and hard-earned.
Best of all, this book flows. Like I did with my own contribution to the series, The Rails Way, Barrett
has made a noble effort to make the book readable from front to back. Each chapter builds on the previous
one, so that by the time you finish it you can go out and land a high-paying job as a Data Specialist! Well,
your mileage may vary, but you think I’m joking? Don’t tell anyone, but I got my first professional job as a
Java programmer back in 1996 after reading Java in 21 Days!
Let me know if you try it. And let’s see here, let me know if you know anyone that can write some of
these series books from my list, especially Domain Specific Languages in Ruby. That would be truly
epic!
Obie Fernandez
Brooklyn, NY
June 2016
Preface
I love data.
I have spent several years working with a lot of different types of data. Sometimes you control the data
collection, and sometimes you have to hunt down the data you need. Sometimes the data is clean and
orderly, and sometimes it requires a lot of work to clean it up.
What makes data interesting to me is that each project is different. They each ask something different of
you to bring their stories to life. As I worked through these visualizations I was reminded just how many
different skills and techniques come into play. Everything is aimed at a singular goal, though—to cut
through the clutter and let the data say what it has to say.
That is what this book is about—giving data a voice.
Audience
This book focuses on looking at data from the perspective of a web developer. More specifically, I’ll
speak from the perspective of a developer writing Ruby on Rails apps.
This book will make use of the following languages and tools: • Ruby on Rails (Rails 4.2.6)
• jQuery
• D3.js
• Leaflet.js
• PostgreSQL
• PostGIS
Do not worry if you are not too comfortable with something on that list or even anything on the list. I
will guide you through the process so that by the end of the book you feel comfortable with all of them.
Organization
I wrote with the intent of each chapter building on the previous chapter. You can see in “Structure and
Content” how the sections and chapters are broken up. My goal for readers who want to read the book
linearly from cover to cover is that by the end you feel like you have a solid foundation for working with
data, including geospatial data.
You could also approach this book from the perspective of wanting to see how to do something. In that
case you could look to the Index to find what you are looking for. You could also look at the
“Supplementary Materials” to see the commits for the three applications that are built through the course
of the book. Feel free to look through the source code and play with it
(https://github.com/DataVizToolkit/).
Supplementary Materials
Throughout the course of this book we will build three Rails applications. The source code is available
so that you verify that you are following along correctly. The applications are broken up as follows:
U
U NTO blest Melancholy's house one happy day
I took my way:
Into a chamber was shown, whence could be seen
Her flowerless garden, dyed with sunlit green
Of myrtle, box, and bay.
'Nay, I am bound.
And your cry faints out in my mind.
Peace not on earth have I found,
Yet to earth am resigned.
Cease thy shrill mockery, Voice,
Nor answer again.'
'O Master, thick cloud shuts thee out
And cold tempests of rain.'
MAERCHEN
'Landsman, landsman,
Lynx nor kestrel
Ne'er shall descry from
Ocean steep
That midnight-stealing, high-pooped galliass, Sleep.'
I N sea-cold Lyonesse,
When the Sabbath eve shafts down
On the roofs, walls, belfries
Of the foundered town,
The Nereids pluck their lyres
Where the green translucency beats,
And with motionless eyes at gaze
Make minstrelsy in the streets.
I
I N a dense wood, a drear wood,
Dark water is flowing;
Deep, deep, beyond sounding,
A flood ever flowing.
T
T HE mild noon air of Spring again
Lapped shimmering in that sea-lulled lane.
Hazel was budding; wan as snow
The leafless blackthorn was a-blow.
O restless thought
Contented not. With 'Why' distraught.
Whom asked you then your riddle small?—
'If hither came no man at all
(To Colin)
C
C RASHED through the woods that lumbering Coach. The dust
Of flinted roads bepowdering felloe and hood.
Its gay paint cracked, its axles red with rust,
It lunged, lurched, toppled through a solitude
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
textbookfull.com