Commit Graph

94 Commits

Author SHA1 Message Date
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
Archana BS b5a860b234 analytics: Populate messages_in_stream:is_bot:day in dev. 2018-03-04 13:17:00 -08:00
Rishi Gupta 1d581a9c6e nagios: Add nagios check for analytics state.
This should help us detect issues where the analytics cron jobs aren't
running properly.

The cron/nagios part of the implementation done by tabbott.
2018-02-09 16:36:05 -08:00
rht 01885cdedc analytics: Use Python 3 syntax for typing (final). 2017-11-22 12:16:59 -08:00
rht d1689b5884 analytics: Use python 3 syntax for typing. 2017-11-17 13:16:49 -08:00
Tim Abbott 2b43a0302a python: Sort imports in smaller apps. 2017-11-15 15:55:49 -08:00
rht 51c1a6dfc9 analytics: Text-wrap long lines exceeding 110.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-10 16:22:00 -08:00
derAnfaenger 19bc55aa45 Fix various typos.
The typos and their corrections were found with the
aid of https://github.com/lucasdemarchi/codespell.
2017-11-09 16:26:38 +01:00
rht fa09076ec9 analytics/management: Remove unused imports (F401). 2017-11-07 16:37:09 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht 2f3ae84e5a py3: Remove all `__future__ import division`. 2017-10-17 23:09:12 -07:00
Rishi Gupta e31758c257 analytics: Do not run update_analytics_counts if there are no realms.
Having no realms was not possible before, but will be once system bots are
no longer on a special system realm.
2017-10-05 11:22:06 -07:00
Rishi Gupta 0f31cddf49 analytics: Add management command to clear single stat. 2017-10-05 11:22:06 -07:00
rht dc5136ed96 analytics: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
rht 4494112862 analytics: Remove absolute_import. 2017-09-27 20:20:07 -07:00
rht 74f8a527e4 analytics: Remove print_function. 2017-09-27 18:05:45 -07:00
Greg Price c127630dcf Delete some obsolete usage-stats tools.
These are no longer useful, with our spiffy new analytics framework,
and we haven't in fact been using them for some time, while the
`active-user-stats` cron job does cause regular mail from cron.
Just delete them.
2017-07-31 17:06:15 -07:00
Vishnu Ks b0e4cfd480 analytics: Replace get_user_profile_by email in client_activity. 2017-07-14 13:35:43 -07:00
Aditya Bansal 42b0680ab2 pep8: Add compliance with rule E261 populate_analytics_db.py. 2017-05-31 17:07:15 -07:00
umkay d9b23b39d3 mypy: Fix strict-optional in analytics. 2017-05-26 15:39:39 -07:00
Aditya Bansal 13d9b98c39 pep8: Add compliance with rule E261 to analyze_mit.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 9e11185fe2 pep8: Add compliance with rule E261 to active_user_stats.py. 2017-05-07 23:21:50 -07:00
Rishi Gupta 92978d6fb2 analytics: Fix --utc argument in update_analytics_counts.py. 2017-04-28 16:15:07 -07:00
Rishi Gupta 73ae2abd4e analytics: Add --verbose option to update_analytics_counts.py. 2017-04-28 16:15:07 -07:00
Rishi Gupta dfbeab73b5 analytics: Change update_analytics_counts to only use hour boundaries.
Fixes a recent regression where analytics were not being run on hour
boundaries.

Includes a migration that dumps all the analytics data.
2017-04-28 16:15:07 -07:00
Rishi Gupta f595f4f7f2 analytics: Change Number of Users chart to use realm_active_humans::day.
Previously we showed the total number of users with an active account. This
changes it to show only the number of users that have logged in in the past
two weeks.
2017-04-25 18:35:13 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
hackerkid 55c3d12078 Replace timezone.utc with timezone_utc. 2017-04-16 12:28:56 -07:00
Rishi Gupta 9b661ca91f analytics: Replace CountStat.is_gauge with interval.
Groundwork for allowing stats like "Monthly Active Users".

CountStat.interval is no longer as clean a value as before, so removed it
from views.get_chart_data. It wasn't being used by the frontend anyway.

Removing interval from logger calls in counts.py is not a big loss since we
now include the frequency (which is typically also the interval) in
CountStat.property.
2017-04-14 11:41:07 -07:00
Rishi Gupta f3fc9721f4 analytics: Match client names in populate_analytics_db to populate_db.
Originally, all the client names in populate_analytics_db started with
underscores to make it easy to selectively delete and regenerate them when
re-running populate_analytics_db.

We eventually want to merge populate_analytics_db into populate_db though,
in which case it makes more sense for them to share client names, and not
worry about the case where we run (or re-run) populate_analytics_db
independently of populate_db.
2017-04-12 11:45:15 -07:00
Rishi Gupta 30024d0a8f models: Remove Realm.domain. 2017-03-25 19:55:48 -07:00
hollywoodno dd067c761a analytics: Separate private messages from group private messages.
This makes it possible for our graphs to show the group private
message counts as separate from 1:1 private messages.

Fixes #4102.
2017-03-20 11:46:29 -07:00
Rishi Gupta 51b7677db7 Add RealmAuditLog table and record user activation/deactivation events.
The RealmAuditLog will make it easier for server admins to replay history.
2017-03-04 16:45:44 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 1453a5bfda Change string_id of test zephyr realm from mit to zephyr.
Also changes Realm.is_zephyr_mirror_realm to use string_id=zephyr instead of
domain=mit.edu.
Part of a larger migration away from Realm.domain.
2017-03-04 12:18:01 -08:00
Rishi Gupta 4dc791f393 Clean up timestamps.py and add a test. 2017-03-01 23:03:56 -08:00
Rishi Gupta 562bc6429c Replace datetime.now() with timezone.now() in Django ORM queries.
When you pass a naive datetime to the Django ORM, it uses settings.TIME_ZONE
for the time zone. In the development environment, both settings.TIME_ZONE
and datetime.now() use 'America/New_York', so there is no change in behavior
there. (fromtimestamp with no tz argument uses the same timezone as
datetime.now)

We are soon going to change settings.TIME_ZONE to UTC, so need to remove
naive datetimes from queries to the ORM.
2017-03-01 22:54:28 -08:00
Rishi Gupta 01a4615f6e Change datetime.now to timezone.now in active_user_stats_by_day.
This actually fixes previously broken behavior, since 'date' here gets
turned into the 'day' argument of seconds_active_during_day(day), where
tzinfo is set to UTC.
2017-03-01 22:54:28 -08:00
Rishi Gupta 2b2be8120f Change datetime.now(tz=X) to timezone.now().
datetime.now with a timezone set is equivalent to timezone.now() if it's
never being printed out, but the latter is cleaner and more idiomatic.
2017-03-01 22:54:28 -08:00
Rishi Gupta ce89c64f43 stats.js: Move name_map computation to the backend. 2017-02-10 14:41:18 -08:00
Rishi Gupta a1b1ffe1e4 analytics: Base default views end_time on FillState, not current time. 2017-02-10 14:41:07 -08:00
Tim Abbott e8b0880320 analytics: Log updates to analytics counts. 2017-02-01 17:02:46 -08:00
umkay 76f3d02590 analytics: Add cron job to run analytics jobs.
This adds a cron job to update the Zulip analytics counts, complete
with locking etc.

Substantially tweaked by tabbott.
2017-02-01 17:02:46 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 9640a9e864 lint: Clean up E712 PEP-8 rule. 2017-01-23 21:11:18 -08:00
Rishi Gupta 734ca4644c analytics: Add random_seed argument to generate_time_series_data. 2017-01-17 15:54:57 -08:00
Rishi Gupta 37bdc7c010 analytics: Remove COUNT_STATS['messages_sent:hour'].
Having both messages_sent:hour and messages_sent:is_bot:day is confusing,
since a single messages_sent:is_bot:hour would have a superset of the
information and take less total space. This commit and its parent together
replace the two stats with a single messages_sent:is_bot:hour.
2017-01-17 15:54:57 -08:00
Rishi Gupta b593ac9d7c analytics: Change messages_sent:is_bot to hourly frequency.
In preparation for replacing messages_sent.
2017-01-17 15:54:57 -08:00
Rishi Gupta 68fcb4152f analytics: Remove interval field from *Count tables.
Includes a database migration. The interval field was originally there to
facilitate time aggregation (e.g. aggregate_hour_to_day), but we now do such
aggregations in views code or in the frontend.
2017-01-17 15:54:57 -08:00