Commit Graph

148 Commits

Author SHA1 Message Date
Rishi Gupta 6374596a77 analytics: Add initial fixture for testing views. 2017-01-10 17:48:07 -08:00
Tim Abbott 3f8d4193da lint: Fix % comprehensions being used without a tuple. 2017-01-09 11:45:11 -08:00
Rishi Gupta ac29928d91 Remove domain from analytics management commands. 2017-01-09 11:26:08 -08:00
Rishi Gupta e14f575979 Remove domain from analytics/views.py. 2017-01-09 11:26:08 -08:00
Rishi Gupta 552d626ef2 analytics: Fix FillState.last_modified not being updated.
We were updating FillState with FillState.objects.filter(..).update(..),
which does not update the last_modified field (which has auto_now=True).
The correct incantation is the save() method of the actual FillState
object.
2017-01-08 23:36:34 -08:00
Rishi Gupta 190d320afa analytics: Change CountStat.property from Text to str. 2017-01-08 17:24:51 -08:00
Rishi Gupta a07757c127 analytics/views: Fix query in get_messages_sent_to_realm. 2017-01-08 17:24:51 -08:00
Rishi Gupta f8962d521d analytics: Fix uses of 'interval' in arguments and variable names.
interval refers to a time interval, and frequency refers to something that
semantically means something closer to 'hourly' or 'daily'.

Currently, interval can have values 'hour', 'day', or 'gauge', and frequency
can only have values 'hour' and 'day'.
2017-01-08 17:24:51 -08:00
Rishi Gupta f5899dd14b analytics: Add lib/ function to drop all analytics tables. 2017-01-08 17:24:51 -08:00
Rishi Gupta 73dc904e9c analytics: Move time_range from views.py to lib/time_utils.py 2017-01-08 17:24:51 -08:00
Tommy Ip 28abfca565 analytics: Fix bare except clause. 2017-01-08 16:25:22 -08:00
Rishi Gupta 2b0a7fd0ba Rename models.get_realm_by_string_id to get_realm.
Finishes the refactoring started in c1bbd8d. The goal of the refactoring is
to change the argument to get_realm from a Realm.domain to a
Realm.string_id. The steps were

* Add a new function, get_realm_by_string_id.

* Change all calls to get_realm to use get_realm_by_string_id instead.

* Remove get_realm.

* (This commit) Rename get_realm_by_string_id to get_realm.

Part of a larger migration to remove the Realm.domain field entirely.
2017-01-04 17:12:23 -08:00
Rishi Gupta 605361ec86 makemessages: Fix string with unnamed arguments in analytics/views.py. 2016-12-30 16:52:24 -08:00
Rishi Gupta 9e5325a164 Add /stats page with basic stats graph.
Adds a new url route and a new json endpoint.
2016-12-29 14:20:13 -08:00
Rishi Gupta 31efe858ef Clean up imports in analytics/views.py. 2016-12-29 14:20:13 -08:00
Rishi Gupta 717afcb408 Remove calls to get_realm in preparation for its deprecation.
Also removes two calls to email_to_domain.
2016-12-26 17:53:32 -08:00
Rishi Gupta c7c0e36508 analytics: Add InstallationCount checks to prototype TestCountStat.
Was enabled by commit 41e8ee3 where we moved TIME_ZERO to before the realms
created by populate_db.py.

Also removes the stub for TestAggregates, since the remaining thing to be
tested was the aggregation from RealmCount to InstallationCount, and the end
to end checks provided by the TestCountStat tests should be sufficient.
2016-12-20 12:03:23 -08:00
Rishi Gupta dbc94d0fc0 analytics: Remove test for no longer supported behavior.
In a previous design, there was no FillState table, and one could run any
CountStat at any time. This is no longer supported.

This test was making sure that if one ran a CountStat at a certain hour, and
then ran it at a previous hour, the old rows would still be there.
2016-12-20 12:03:23 -08:00
Rishi Gupta e09aaf1020 analytics: Remove tests that will be subsumed by TestCountStats. 2016-12-20 12:03:23 -08:00
Rishi Gupta 6748b72ccc analytics: Remove tests now covered by test_active_users_by_is_bot. 2016-12-20 12:03:23 -08:00
Rishi Gupta 2211b8b102 analytics: Change count_message_by_stream to join on UserProfile.
It seems unlikely we will need count_message_by_stream without the
UserProfile table in the future, so write count_message_by_stream_and_is_bot
in the usual query form and replace count_message_by_stream with it.
This also has the benefit of shortening our list of "special case" queries
from two to one.

The pathways of the removed test will be covered more thoroughly in the new
TestCountStats tests.
2016-12-20 12:03:23 -08:00
Rishi Gupta 6992f9784c analytics: Update TestCountStat prototype. 2016-12-20 12:03:23 -08:00
Rishi Gupta c6a6c871ee analytics: Change TIME_ZERO in tests to be in the past. 2016-12-20 12:03:23 -08:00
Rishi Gupta d95fb33d8d analytics: Add subgroups to unicode representations in models.py. 2016-12-20 12:03:23 -08:00
Rishi Gupta f34af0896d analytics: Add subgroup argument to assertCountEquals. 2016-12-20 12:03:23 -08:00
Rishi Gupta 31cf8db28c analytics: Allow assertCountEquals to work on InstallationCount. 2016-12-20 12:03:23 -08:00
Rishi Gupta 93a10a475a counts.py: Fix count_message_type_by_user_query. 2016-12-15 16:02:12 -08:00
Rishi Gupta 4f3e1b2ece analytics/lib/counts.py: Fix messages_sent_to_stream:is_bot.
Adds a new query.
2016-12-15 16:02:12 -08:00
Rishi Gupta 87b47ec283 analytics: Add __unicode__ method to the CountStat object. 2016-12-15 16:02:12 -08:00
reyha 82e32ad255 Access realm by `string_id` in management commands.
`Realm.string_id` replaces 'Realm.domain'
in the management commands.

Fixes #2325.
2016-12-14 10:38:03 -08:00
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
bulat22101 adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
AZtheAsian 1ba150fa85 pep8: Fix E203 violations 2016-12-01 20:37:57 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Rishi Gupta 4b183cd526 domain migration: Remove several instances of get_realm.
Remove the easy to remove instances of get_realm.
2016-11-26 15:19:56 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Umair Khan 7d51efe9a1 Django 1.10: Fix dummy data for count stat.
Django 1.10 checks the foreign key constraints as part of the testing
suite so we need to create test data which passes validation tests.
2016-11-14 16:09:12 -08:00
umkay dc8463e09c analytics: Remove incorrect filter args for stat.
The filter args dictionary applies to the X table in a count X by Y query,
which in this case is the zerver_message table. This stat had an incorrect set
of arguments meant for the zerver_userprofile table.
2016-11-10 12:25:21 -08:00
Umair Khan d837753d4b Django 1.10: Update analytics urls. 2016-11-10 16:20:03 +05:00
Umair Khan 682aa1f298 Django 1.10: Use add_argument for options in BaseCommand. 2016-11-04 10:20:23 -07:00
Umair Khan b140236fcf Django 1.10: Do not use patterns function. 2016-11-04 10:06:00 -07:00
umkay e6ac8c3543 analytics: Add extra count stats.
Fill in remaining countstats in counts.py for our intended use cases.
2016-11-03 16:50:39 -07:00
umkay 298890d125 analytics: Rename count stats and associated properties.
Our current naming convention is getting unwieldy. The subgroup now goes
on the right side of the colon.
2016-11-03 16:50:39 -07:00
umkay 5490442580 analytics: Replace all joins in raw SQL with natural joins.
We alter the behavior of our queries to no longer write rows with 0 counts
to the db, and pad with 0s in the related views code. As a result we are
also able to combine the where and join clause conditions in the sql
queries. This new behavior is also updated in our tests.
2016-11-03 16:50:39 -07:00
Rishi Gupta db0e509422 do_create_realm: Replace domain argument with string_id.
Turns string_id into a required argument, and domain into an optional
argument.
2016-11-02 22:46:34 -07:00
Rishi Gupta 9ef8536cc6 models.Realm: Require Realm.string_id to be non-NULL.
Adds a database migration, adds a new string_id argument to the management
realm creation command, and adds a short name field to the web realm
creation form when REALMS_HAVE_SUBDOMAINS is False.
2016-11-02 22:46:34 -07:00
umkay 5e5a0d4db9 analytics: Add user-level count query for messages sent to {PMs, streams}.
Adds a count_X_by_Y_query to counts.py, similar in spirit to a
count_recipient_by_user query, where we would join on the Message,
Recipient, and UserProfile table. Here, we also join on the Stream table in
order to distinguish private and public streams, and we merge the counts for
PM and Huddle type messages into a single subgroup.
2016-11-01 17:00:43 -07:00