Commit Graph

164 Commits

Author SHA1 Message Date
Tim Abbott 0b60f11cf8 analytics: Display Zulip Electron app as the main desktop app.
This should make the /stats data significantly clearer.
2017-07-07 18:31:47 -07:00
Tim Abbott 31caab4229 analytics: Rename 'new iOS app' to 'Mobile app'. 2017-07-07 18:31:13 -07:00
Aditya Bansal 30dfb54c65 pep8: Add compliance with rule E261 to analytics/views.py. 2017-05-31 17:07:15 -07:00
umkay d9b23b39d3 mypy: Fix strict-optional in analytics. 2017-05-26 15:39:39 -07:00
Rishi Gupta ee16abaf3b analytics: Fix sort function in views.sort_client_labels.
sort_client_labels sorts first by total, and then to ensure deterministic
outcomes, sorts (reverse) alphabetically by label.

Fixes regression introduced in 0c0e539.
2017-05-09 11:32:35 -07:00
Rishi Gupta f595f4f7f2 analytics: Change Number of Users chart to use realm_active_humans::day.
Previously we showed the total number of users with an active account. This
changes it to show only the number of users that have logged in in the past
two weeks.
2017-04-25 18:35:13 -07:00
Maxim Averin 73a1dd63d5 analytics: Refactor legacy 'zulip_internal' decorator.
Rename 'zulip_internal' decorator to 'require_server_admin', add
documentation for 'server_admin', explaining how to give permission
for ./activity page.

Fixes: #1463.
2017-04-22 11:42:02 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Rishi Gupta 9b661ca91f analytics: Replace CountStat.is_gauge with interval.
Groundwork for allowing stats like "Monthly Active Users".

CountStat.interval is no longer as clean a value as before, so removed it
from views.get_chart_data. It wasn't being used by the frontend anyway.

Removing interval from logger calls in counts.py is not a big loss since we
now include the frequency (which is typically also the interval) in
CountStat.property.
2017-04-14 11:41:07 -07:00
Rishi Gupta b555191ee7 analytics: Change subgroup and labels lists to dictionary in views.py.
A dict is more semantically appropriate than two lists with an assertion
that they be the same length.
2017-04-13 12:37:47 -07:00
Rishi Gupta f3fc9721f4 analytics: Match client names in populate_analytics_db to populate_db.
Originally, all the client names in populate_analytics_db started with
underscores to make it easy to selectively delete and regenerate them when
re-running populate_analytics_db.

We eventually want to merge populate_analytics_db into populate_db though,
in which case it makes more sense for them to share client names, and not
worry about the case where we run (or re-run) populate_analytics_db
independently of populate_db.
2017-04-12 11:45:15 -07:00
Rishi Gupta 9f60dd8387 analytics: Send zeros for data.user.bot in Messages Sent Over Time.
It will simplify the logic needed to process the "Sent by Me" view in
Messages Sent Over Time in stats.js.

Also, we gzip the data sent from our server, so there is little additional
network usage by doing this.
2017-03-25 14:18:23 -07:00
Tim Abbott 20a7609018 analytics: Rename message count types to use standard Zulip casing. 2017-03-21 00:09:54 -07:00
hollywoodno dd067c761a analytics: Separate private messages from group private messages.
This makes it possible for our graphs to show the group private
message counts as separate from 1:1 private messages.

Fixes #4102.
2017-03-20 11:46:29 -07:00
Tim Abbott 0c0e5397c4 analytics: Fix nondeterministic ordering of labels. 2017-03-20 11:39:08 -07:00
Umair Khan 4442703011 jinja2: No need for custom render_to_response.
Django 1.10 has changed the implementation of this function to
match our custom implementation; in addition to this, we prefer
render().

Fixes #1914 via #4093.
2017-03-17 13:57:34 -07:00
Umair Khan 6511f929cc analytics: Change render_to_response to render.
Related to #4093
2017-03-17 13:52:59 -07:00
Rishi Gupta ebebd04587 analytics: Fix ValueErrors affecting test coverage.
Pathways that only catch internal code errors should use AssertionError so
that they are not included when computing test coverage.
2017-03-14 16:59:54 -07:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 2b2be8120f Change datetime.now(tz=X) to timezone.now().
datetime.now with a timezone set is equivalent to timezone.now() if it's
never being printed out, but the latter is cleaner and more idiomatic.
2017-03-01 22:54:28 -08:00
Rishi Gupta d6ce017a58 analytics: Minor cleanup of views.py. 2017-02-11 14:51:01 -08:00
Rishi Gupta 480fc0874b analytics: Break ties deterministically in sort_client_labels. 2017-02-11 14:51:01 -08:00
Tim Abbott f944ac8902 mypy: Fix incorrect annotation for by_used_time. 2017-02-10 23:53:44 -08:00
Rishi Gupta 19d1fc6223 stats: Pass user data to the frontend for messages sent over time. 2017-02-10 14:41:18 -08:00
Rishi Gupta 68a7f91022 stats: Add a fixed display order to summary charts.
API: Adds a "display_order" to the response, which is a suggested order of
importance for the clients or recipient types respectively.

frontend: Changes messages_sent_by_{client,recipient_type} to use a fixed
order for any given user.
2017-02-10 14:41:18 -08:00
Rishi Gupta cf3ae2eafe stats: Turn messages_sent_by_client into a bar chart.
Also includes a number of changes to messages_sent_by_recipient_type that
were convenient to do at the same time, since the two charts share a lot of
code.
2017-02-10 14:41:18 -08:00
Rishi Gupta ce89c64f43 stats.js: Move name_map computation to the backend. 2017-02-10 14:41:18 -08:00
Rishi Gupta a1b1ffe1e4 analytics: Base default views end_time on FillState, not current time. 2017-02-10 14:41:07 -08:00
Rishi Gupta 6ab31d1bac analytics: Move time computation to later in get_chart_data. 2017-02-10 14:40:14 -08:00
Tim Abbott ec52322ae1 stats: Include Zulip and realm name in heading. 2017-02-07 11:22:57 -08:00
Tim Abbott 6c4eaf3d14 analytics: Map client names to user-facing versions.
This makes the pie charts on /stats more readable.
2017-02-05 22:19:10 -08:00
Amy Liu 0a39e354dc analytics: Add graphs of usage statistics on /stats.
This adds a frontend for the analytics system we've had for a few
months, showing several graphs of the data in Zulip.

There's a ton more that we can do with this tooling, but this initial
version is enough to provide users with a pretty good experience.

Fixes #2052.
2017-01-31 22:18:54 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 9cc83f87fc lint: Clean up E241 PEP-8 rule. 2017-01-23 21:21:14 -08:00
Rishi Gupta 29799d93c6 analytics/views.py: Always return time series data for stats.
Makes a number of simplications to the analytics views code. The main one is
that we now return the entire data series, even if the data is eventually
going to go into a pie chart. This was prompted by us wanting several
different pie charts for each stat (one for last 30 days, one for all time,
etc), but I think it is also a more natural API. The total amount of data
being sent for the pie charts now is maybe half of what is being sent for
our single 'hourly' stat, or maybe up to 10,000 ints per year the
organization has been around.

The other big change is that the data being sent back is now always explicit
about whether it is data about the realm (stored in data['realm'], or data
about the user (stored in data['user']).
2017-01-19 17:44:17 -08:00
Rishi Gupta b593ac9d7c analytics: Change messages_sent:is_bot to hourly frequency.
In preparation for replacing messages_sent.
2017-01-17 15:54:57 -08:00
Rishi Gupta a8f2ebb443 analytics: Include interval in COUNT_STATS property names. 2017-01-17 15:54:57 -08:00
Rishi Gupta ecfc336a15 analytics: Add views for remaining /stats graphs. 2017-01-17 15:54:57 -08:00
Rishi Gupta 73c0c4c52e analytics/views.py: Increase efficiency of get_time_series_by_subgroup.
Not sure if this would actually be a performance problem in practice, but
this was originally making a database query for each subgroup (instead of
just a single query getting data for all the subgroups).

Also removed the filter against the interval column, which will soon not be
needed (interval will be uniquely determined by the property).
2017-01-17 15:54:57 -08:00
Rishi Gupta d873902755 analytics/views.py: Refactor get_messages_sent_by_humans_and_bots.
Refactor out the reusable parts, since we're about to add several more
views.
2017-01-17 15:54:57 -08:00
Rishi Gupta 3a72b5cda9 analytics: Rename messages_sent_to_realm.
Several additional stats in the pipeline that also relate to messages sent
to the realm.
2017-01-17 15:54:57 -08:00
Rishi Gupta f375caed46 /activity: Fix URL route for analytics.views.get_realm_activity.
analytics.views.get_realm_activity was taking a 'realm_str', but the URL
route was expecting a 'realm'. Changed the URL route to take a 'realm_str'.
2017-01-12 15:21:06 -08:00
Tim Abbott 3f8d4193da lint: Fix % comprehensions being used without a tuple. 2017-01-09 11:45:11 -08:00
Rishi Gupta e14f575979 Remove domain from analytics/views.py. 2017-01-09 11:26:08 -08:00
Rishi Gupta a07757c127 analytics/views: Fix query in get_messages_sent_to_realm. 2017-01-08 17:24:51 -08:00
Rishi Gupta f8962d521d analytics: Fix uses of 'interval' in arguments and variable names.
interval refers to a time interval, and frequency refers to something that
semantically means something closer to 'hourly' or 'daily'.

Currently, interval can have values 'hour', 'day', or 'gauge', and frequency
can only have values 'hour' and 'day'.
2017-01-08 17:24:51 -08:00
Rishi Gupta 73dc904e9c analytics: Move time_range from views.py to lib/time_utils.py 2017-01-08 17:24:51 -08:00
Tommy Ip 28abfca565 analytics: Fix bare except clause. 2017-01-08 16:25:22 -08:00
Rishi Gupta 605361ec86 makemessages: Fix string with unnamed arguments in analytics/views.py. 2016-12-30 16:52:24 -08:00
Rishi Gupta 9e5325a164 Add /stats page with basic stats graph.
Adds a new url route and a new json endpoint.
2016-12-29 14:20:13 -08:00
Rishi Gupta 31efe858ef Clean up imports in analytics/views.py. 2016-12-29 14:20:13 -08:00
Rishi Gupta 717afcb408 Remove calls to get_realm in preparation for its deprecation.
Also removes two calls to email_to_domain.
2016-12-26 17:53:32 -08:00
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Rishi Gupta 4b183cd526 domain migration: Remove several instances of get_realm.
Remove the easy to remove instances of get_realm.
2016-11-26 15:19:56 -08:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Eklavya Sharma 71e613424b Fix annotations clashing with UserProfile's model fields. 2016-06-13 20:01:01 +05:30
Hyunchel Kim f226456675 Add type annotations for analytics/views.py.
Type of parameter for function `is_recent`(line no.812) is `datetime`.
MyPy errors out, however, when the parameter is defined as `datetime`.
To get around, type `Any` is used.
2016-06-05 15:04:24 -07:00
Tim Abbott efd24b374e analytics: Fix cnts variable reuse with different type.
Found using mypy.
2016-05-12 14:07:32 -07:00
Umair Khan 5359e6b0d4 Convert Zulip to use Jinja2 templates.
This results in a substantial performance improvement for all of
Zulip's backend templates.

Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.

Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`

Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
  instead of `RequestContext` object. There are two reasons to do this:
    1. `RequestContext` is not compatible with Jinja2
    2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
  includes a custom backend and a template renderer, compressors for js
  and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.

Fixes #620.
2016-05-09 09:55:18 -07:00
Umair Khan 6a0c7fec72 analytics: Add `at_risk_count` to Totals row in realm summary.
This fixes reading from an unset value in realm_summary_table, which
is fine with the Django template engine but will be problematic with
jinja2.
2016-05-07 17:30:06 -07:00
Tim Abbott 54022ac204 Fix unnecessary whitespace between , and ). 2016-05-04 14:16:53 -07:00
Ashish 6356584f84 Replace /json/update_pointer with REST style route. 2016-04-11 21:38:23 -07:00
Tim Abbott a1b306f9ce Finish purging 'fromt typing import *' from Zulip codebase. 2016-04-07 14:11:21 -07:00
Tim Abbott b8c82d5b43 Add PEP-484 type annotations to analytics/. 2016-04-03 15:40:23 -07:00
Tim Abbott 2436ad19ba analytics: Cleanup confusingly type-variable all_records. 2016-02-03 19:29:07 -08:00
Tim Abbott 1f44417fc1 Switch to using Python 3 style division everywhere.
Also add testing for this to our Python 3 compatibility test suite.
2016-01-26 21:09:43 -08:00
Tim Abbott 607eedfc25 Apply Python 3 futurize transform libmodernize.fixes.fix_zip. 2015-11-01 09:35:06 -08:00
Tim Abbott f7878a61e1 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six. 2015-11-01 09:35:06 -08:00
Tim Abbott cd6f8e9191 Apply Python 3 futurize transform libmodernize.fixes.fix_map. 2015-11-01 09:35:05 -08:00
Tim Abbott b3ac668779 Apply Python 3 futurize transform libmodernize.fixes.fix_filter. 2015-11-01 09:26:16 -08:00
Tim Abbott 71a06d58de Convert uses of Realm.objects.get() to get_realm().
get_realm is better in two key ways:
* It uses memcached to fetch the data from the cache and thus is faster.
* It does a case-insensitive query and thus is more safe.
2015-10-15 09:16:58 -04:00
Tim Abbott e29c473077 Simplify analytics code to not filter certain low-interest users/realms.
(imported from commit 2dcf2e50b65c8b96d893cbe7dcdbbe652e6a90ff)
2015-09-19 23:42:28 -07:00
Steve Howell 9224d4b521 Fix comment in meets_goal() function in activity reports.
(imported from commit b07a6059b4fc9ecb609881e7b619ac10cafb9e22)
2014-02-03 13:30:49 -05:00
Tim Abbott 85381a62b9 analytics: Show zulip.com and mit.edu counts on /activity.
To make this give accurate numbers, we need to filter out the
automated traffic from Zephyr mirroring and our internal monitoring.

(imported from commit 83642bc9a9d8d01dd9dc5dc7b3e3dee6c9705162)
2014-02-03 12:38:28 -05:00
Steve Howell 13d015f81e Show /api/v1/send_message traffic in /activity reports.
(imported from commit 5230f92c1376a914099e4c4c883f1bee5d776cad)
2014-01-21 11:23:45 -05:00
Steve Howell b880b9426f Handle Realm.DoesNotExist error in /realm_activity.
(imported from commit 8455b840563fe48ff33b3bcc33f442a7cdd6a2ff)
2014-01-10 21:39:04 -05:00
Steve Howell 8ccf3ad36a Count all queries for mobile apps in /activity.
(imported from commit 1ea200fc8d2432af0c66d501c0fc8a226eb11194)
2014-01-10 21:38:57 -05:00
Steve Howell 7f668463ac Add an ZulipiOS tab to /activity.
(imported from commit 09f609e340a8fb7838e74d558eb3e397ea5f1f7b)
2014-01-10 21:38:57 -05:00
Steve Howell 0b9eb74f93 Remove Pure API tab from /activity.
(imported from commit 32ec22de7746637a18c98f93903661851e367bc8)
2014-01-10 21:38:57 -05:00
Steve Howell 26548da779 Use calendar days for /activity message history.
(imported from commit f4d331e8384ff2663fb6aab7170482f057d7403f)
2014-01-06 11:50:15 -05:00
Steve Howell 0c5f543433 Exclude totals row from # of active sites.
(imported from commit 80f38d8f24337244d7a8e1eb409aaed08dd772be)
2013-12-23 09:57:51 -05:00
Steve Howell fcbc7e1516 Do not count wdaher.com in report.
(imported from commit 61a17c3addc0893282c55135261a0c2ffe797273)
2013-12-23 09:57:50 -05:00
Steve Howell ea1fefa5b5 Show message history in reverse chronological order.
(imported from commit 6237369793044cec7ae8ffec42aaac5d443e2ce1)
2013-12-19 14:03:55 -05:00
Steve Howell 986c46d417 Color code "Messages sent"
(imported from commit d648bfbcdd054a57fb6b39d14d0579819c963c40)
2013-12-18 17:30:47 -05:00
Steve Howell ef089692ed Right-justify numbers on /activity.
(imported from commit 11f167626fc3f1338143fc1cf68a1bb1663a93b9)
2013-12-18 16:38:50 -05:00
Steve Howell 5d85bfbf6b Improve "Messages sent" formatting on /activity.
(imported from commit 891760c116d5a3290b49fb2d75eae0f0b227540d)
2013-12-18 16:34:03 -05:00
Steve Howell ef959a748e Exclude mit.edu from history counts.
(imported from commit eb27f4bb0f6eb07c2d72d4f32fa21995f215254e)
2013-12-18 16:34:03 -05:00
Steve Howell 7cded2a956 Show realm history on the main activity page.
This shows the number of messages sent by humans for the last
eight 24-hour periods, for each realm.  "Messages sent" isn't a
perfect metric of activity, but it's easier to query with our
current data model than certain other statistics.

(imported from commit 9de3c479640a0b9dbc017b245dda21d951f4efa4)
2013-12-18 16:01:44 -05:00
Steve Howell 532fa9fae8 Remove dead code in /activity reports.
(imported from commit ea9ddae01fd5527c6a4ceefaaa5fac091618f544)
2013-12-18 13:42:30 -05:00
Steve Howell 0c1f57815c Add grahite to link to /realm_activity.
(imported from commit 6d2a23ffcca23a65e5aadb2ed8499645e42131e0)
2013-12-18 13:40:07 -05:00
Leo Franchi 1e7a22f14e Replace other non-zerver uses of iPhone client
(imported from commit 0988e2c9bd0499a0711daed97f89aa672776f085)
2013-12-03 14:35:24 -05:00
Steve Howell 14ae79b311 Count /api/*/external/* messages as "sent" in /realm_activity.
This makes it easier to see how many messages are being sent
by webhook bots.  This assumes a 1:1 relationship between
hitting webhook endpoints and sending messages, which is probably
valid enough in the near future.

(imported from commit eb272cd38b9cabd54d317ce2dfdf12099d302fce)
2013-11-25 15:37:19 -05:00
Steve Howell b5707b9c7b Add is_active check to /realm_activity report.
(imported from commit e531446bc73caf7173e4aa1d22b35f3942c9c1f9)
2013-11-24 16:35:19 -05:00
Steve Howell 07d4dd66c4 Show admins on /realm_activity page.
(imported from commit 7d6beb4145b86d95b872b175ea8a567b7ce56d23)
2013-11-18 14:41:37 -05:00
Steve Howell 91a463def0 Show webhook clients in activity pages.
(imported from commit 423df7def53583ab8122a26271068423f03a687c)
2013-11-18 12:58:39 -05:00
Steve Howell 76a1754357 Use from/import style for datetime imports in activity views.
(imported from commit 72772f63f00d1b5e9d5d0ccefb56b6714d1a2370)
2013-11-18 12:36:25 -05:00
Steve Howell 38a7aa2255 Highlight recently active users in /realm_activity.
(imported from commit 307ed9c17ea1055f40cd983d5c2b2f308203f3f7)
2013-11-18 12:36:24 -05:00
Steve Howell ae77b98155 DRY up East Coast time formatting.
(imported from commit b5b5787816c2dc641bbe4cfb351f4ff765e44969)
2013-11-18 12:36:24 -05:00