Commit Graph

236 Commits

Author SHA1 Message Date
Vishnu KS 905f3d2d23 stats: Don't show analytics data unavailable error for new realms.
The value of start would be set to realm creation date or installation
date unless a value is explicitly provided by the user. We don't
want to show analytics data missing error in both these cases if
start is less than 24.5 hours from timezone_now() since some of the
stats are populated by a daily cron job that may take a couple of
minutes to run.

The same case applies if the value of the start is passed in the
request.
2020-04-03 11:51:20 -07:00
Vishnu Ks a26b379a14 support: Send confirmation email on realm activation. 2019-12-02 09:51:45 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Vishnu KS ec955f8f78 support: Show confirmation links in search.
Fixes #13060 #12784
2019-10-21 16:56:50 -07:00
Vishnu KS 139ebf387b support: Pass various realm functions as template context.
We currently have code to calculate the value of realm_icon_url,
admin_emails and default_discount in two diffrent places. With
the addition of showing confirmation links it would become three.
The easiest way to deduplicate the code and make the view cleaner
is by doing the calculations in template. Alternatively one can
write a function that takes users, realms and confirmations as
arguments and sets the value of realm_icon_url, admin_emails and
default_discount appropriately in realm object according to the
type of the confirmation. But that seems more messy than passing
the functions directly to template approach.
2019-10-21 16:52:46 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 490ce993f4 analytics: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Wyatt Hoodes 97c10b35c2 analytics/views: Remove redundant check of `if end is None`. 2019-07-31 12:15:57 -07:00
Rishi Gupta 9fc791926b support: Use delivery_email instead of email.
Structurally, these pages want email addresses one can actually send
emails to.
2019-07-21 14:47:15 -07:00
Tim Abbott 4eb9d67b70 models: Extract get_human_admin_users function.
This function is an alternative to get_admin_users that we use in all
places where we explicitly want only human administrative users (not
administrative bots).  The following commits will rename
get_admin_users for better clarity.
2019-06-20 14:32:30 -07:00
Mohit Gupta db3d81613b decorator: Refactor @require_non_guest_human_user decorator.
Rename @require_non_guest_human_user to @require_member_or_admin.
This is a refactor commit prior to introduction of Administrator
Bots.
2019-06-18 17:11:58 -07:00
Vishnu Ks 14e582fb59 support: Add functionality to copy admin emails.
Also renamed a bunch of functions in test_views for better
readability.
2019-06-14 10:19:50 -07:00
Tim Abbott 1d44fd724b audit log: Log which server admin deactivated a realm too. 2019-05-08 15:09:48 -07:00
Rishi Gupta 0218da64f1 support: Update success message for plan type change. 2019-05-08 15:09:48 -07:00
Rishi Gupta 98da11c558 support: Rename deactive to deactivated. 2019-05-08 15:09:48 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Vishnu Ks 6c58603eaf support: Add support for scrubbing realm. 2019-05-06 20:12:54 -07:00
Vishnu Ks f6203f068b support: Add support for activating and deactivating realm. 2019-05-06 20:12:48 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Vishnu Ks 8eeb8280b4 activity: Create interface for doing support operations.
This should grow into a tool that makes it much easier to do common
organization management tasks without using a manage.py shell.
2019-03-11 12:01:11 -07:00
Vishnu Ks 42de9a0c71 analytics: Extract out the function for getting plan name. 2019-03-11 12:01:11 -07:00
Tim Abbott c41bfcb9e0 Revert "activity: Change definition of active site."
This reverts commit 9f9b7cb991.

This commit made the page not perform well enough to load.
2019-02-13 14:52:13 -08:00
bartek 9f9b7cb991 activity: Change definition of active site.
Signed-off-by: bartek <bartek.jachowicz@gmail.com>

Edits by Rishi Gupta <rishig@zulipchat.com>

Fixes: #10432
2019-02-12 16:36:44 -08:00
Anders Kaseorg c2cb804ebe analytics/views.py: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:43:58 -08:00
Tim Abbott 13b9e90a40 analytics: Fix importing zilencer.
In ebdd55814c, we added zilencer imports
without using the proper mocking procedure for when zilencer is not enabled.

This whole setup is a mess and probably we should enable zilencer
unconditionally in a future version.
2019-02-02 17:18:57 -08:00
Tim Abbott 96f096f38e analytics: Add section for remote Zulip servers. 2019-02-02 16:55:12 -08:00
Tim Abbott ebdd55814c analytics: Add support for graphing remote analytics data.
Combined with recent work on submitting data, one can now view the
high-level usage numbers for chat.zulip.org on zulipchat.com's
dashboards.
2019-02-02 16:55:12 -08:00
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Rishi Gupta 91b02373dc activity: Rename standard free to open source.
"standard free" looks too similar to "standard". Also it makes it hard to
Ctr+f for "standard".
2018-11-16 21:03:37 -08:00
Vishnu Ks 2e04cdbe5e billing: Show estimated subscription revenue on /activity.
[Substantial edits by Rishi Gupta]
2018-11-16 13:30:16 -08:00
Vishnu Ks 9d398ee335 activity: Move the total row from last to first. 2018-11-16 09:22:32 -08:00
Yashashvi Dave 02a5849d4c statistics: Guest user can't access realm statistics.
Don't allow guest user to access realm statistics from
UI or at API level.

Fixes part of #10749.
2018-11-02 11:43:09 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Vishnu Ks 6fcb095e70 activity: Show plan_type in activity page. 2018-08-21 18:15:14 -07:00
Nikhil Kumar Mishra fa9d79e203 stats: Add 1 day actives and total users to number of users chart. 2018-05-20 10:56:16 -07:00
Rishi Gupta 66a589c7a7 stats: Extend get_chart_data to support charts with multiple CountStats. 2018-05-20 10:56:16 -07:00
Rishi Gupta 08bf0a66b8 stats: Refactor the get_time_series_by_subgroup calls in get_chart_data.
This code is going to end up pretty complex -- each stat has multiple levels
of aggregation (UserCount, RealmCount, InstallationCount), and refinement
(subgroups), and soon we'll have charts that take data from multiple stats
as input.

Not sure what the best way to present it is, but hopefully this simplifies
it a bit.
2018-05-20 10:56:16 -07:00
Rishi Gupta 1af7fc7344 stats: Add /stats/installation. 2018-05-18 15:12:36 -07:00
Rishi Gupta 2fe3fba6ce stats: Rename data.realm to data.everyone.
We use "Everyone" for the button labels already.

Soon we'll support "Everyone" meaning either the installation or the realm,
depending on the URL route used to access the stats.
2018-05-18 15:12:36 -07:00
Rishi Gupta af758755bd stats: Rename target_realm_name to target_name in stats.html. 2018-05-18 15:12:36 -07:00
Rishi Gupta e099959a41 stats: Move API route computation from frontend to backend.
Will make it easier to extend to additional routes.
No changes in behavior.
2018-05-18 15:12:36 -07:00
Yago González 184bd8304e i18n: Tag missing strings for translation. 2018-05-12 16:44:56 -07:00
Aditya Bansal 5adf983c3c analytics: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Shubham Dhama 03a2a9c792 activity: Add realm stats link to "realm activtiy table". 2018-04-18 11:07:00 -07:00
Shubham Dhama b26c38bc47 analytics: Make stats of all realms accessible to server admins.
In this commit:
Two new URLs are added, to make all realms accessible for server
admins. One is for the stats page itself and another for getting
chart data i.e. chart data API requests.
For the above two new URLs corresponding two view functions are
added.
2018-04-18 11:06:50 -07:00
Tim Abbott 6bbce451bc analytics: Fix a minor mypy error. 2018-02-12 11:57:47 -08:00
Aman Jain a0b58b1560 activity: Add a link to copy list of realm admins.
Tweaked by tabbott to do the database queries properly.

This should help user to copy realm admin emails in a go.
Fixes: #7885
2018-02-12 08:55:45 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
Greg Price 20b2c11830 activity: Show the time the data is from.
I've wanted this when looking at a tab from the day before.
Also provides the date and time in UTC, which is handy for
interpreting some of the data.

Pretty sure this is not the world's cleanest way to do this in the
front-end code.  It'll do for now.
2017-12-12 15:30:03 -08:00
Greg Price bdea0960de activity: Supply a missing field.
In prod this has no effect, but in dev we render a warning here,
which makes the table not look right.  Fix it.
2017-12-12 15:30:03 -08:00
Greg Price d36b1cd2d7 activity: Correct description of message-history figures.
This explains why the first number was usually the smallest!
2017-12-12 15:30:03 -08:00
Greg Price 34ee019b20 activity: Consistently define "user" as non-deactivated, non-bot. 2017-11-30 20:43:46 -08:00
Greg Price 0ebfc2fb5c activity: Highlight recently-created realms. 2017-11-30 20:43:46 -08:00
Greg Price 8de34c93dd activity: Show each realm's creation date.
I'd rather have something rich and fancy like a sparkline of
activity... but this is a lot quicker to implement.
2017-11-30 20:43:46 -08:00
Greg Price 2c9b698cb6 activity: Replace "at-risk users" with WAU.
Substantively, this makes the table more readable by grouping users
into expanding sets by level of activity: active in last day, active
in last week, have an account at all.  The class "active in last week",
as opposed to "active in last week but not in last day", makes more
natural comparisons both between realms and for one realm through time,
and it's less sensitive to the details of our definitions.

This also makes the terminology more standard.  We already made that
change in the display, in the previous commit; as we go through the
logic here, we adjust the terminology in the code too.
2017-11-30 20:43:46 -08:00
rht 01885cdedc analytics: Use Python 3 syntax for typing (final). 2017-11-22 12:16:59 -08:00
rht d1689b5884 analytics: Use python 3 syntax for typing. 2017-11-17 13:16:49 -08:00
Tim Abbott 2b43a0302a python: Sort imports in smaller apps. 2017-11-15 15:55:49 -08:00
rht 6cce0e346e refactor: Remove six.moves.filter import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht 549a26860f refactor: Remove six.moves.range import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
Tim Abbott 732a5bae84 analytics: Fix checks for pointer update route. 2017-11-04 19:27:00 -07:00
Tim Abbott 1cd017288d views: Fix imports of REQ/has_request_variables from the wrong place.
These were never in zerver/decorator.py, and so it makes sense to
import them zerver/lib/request.py, mostly for ease of finding things.
2017-10-27 15:07:31 -07:00
Rishi Gupta 0c2b4d22a7 analytics: Convert datetimes coming from the API into UTC.
Previously, entering a non-UTC end time for a daily stat would give you
incorrect results. This is because:
* All daily stats are collected at and have end_times in the database in
  midnight UTC.
* For daily stats, time_range returns a list of datetimes at midnight in the
  timezone of its end argument. These datetimes are the only ones we look
  for when looking for rows corresponding to the stat in the database.
* Previously, we passed on the end argument from the API to time_range,
  without modification.
2017-10-05 11:22:06 -07:00
Tim Abbott 575bd0e255 analytics: Update name for old Android app. 2017-10-03 11:59:41 -07:00
rht 4494112862 analytics: Remove absolute_import. 2017-09-27 20:20:07 -07:00
Greg Price 61666a9262 zulip_ops: Delete the long-disused `stats1.zulip.net` config and its dependencies.
This consists of the `zulip_ops::stats` Puppet class, which has apparently
not been used since 2014, and a number of files that I believe were
only used for that.  Also a couple of tiny loose ends in other files.
2017-08-15 17:30:31 -07:00
Tim Abbott 0b60f11cf8 analytics: Display Zulip Electron app as the main desktop app.
This should make the /stats data significantly clearer.
2017-07-07 18:31:47 -07:00
Tim Abbott 31caab4229 analytics: Rename 'new iOS app' to 'Mobile app'. 2017-07-07 18:31:13 -07:00
Aditya Bansal 30dfb54c65 pep8: Add compliance with rule E261 to analytics/views.py. 2017-05-31 17:07:15 -07:00
umkay d9b23b39d3 mypy: Fix strict-optional in analytics. 2017-05-26 15:39:39 -07:00
Rishi Gupta ee16abaf3b analytics: Fix sort function in views.sort_client_labels.
sort_client_labels sorts first by total, and then to ensure deterministic
outcomes, sorts (reverse) alphabetically by label.

Fixes regression introduced in 0c0e539.
2017-05-09 11:32:35 -07:00
Rishi Gupta f595f4f7f2 analytics: Change Number of Users chart to use realm_active_humans::day.
Previously we showed the total number of users with an active account. This
changes it to show only the number of users that have logged in in the past
two weeks.
2017-04-25 18:35:13 -07:00
Maxim Averin 73a1dd63d5 analytics: Refactor legacy 'zulip_internal' decorator.
Rename 'zulip_internal' decorator to 'require_server_admin', add
documentation for 'server_admin', explaining how to give permission
for ./activity page.

Fixes: #1463.
2017-04-22 11:42:02 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Rishi Gupta 9b661ca91f analytics: Replace CountStat.is_gauge with interval.
Groundwork for allowing stats like "Monthly Active Users".

CountStat.interval is no longer as clean a value as before, so removed it
from views.get_chart_data. It wasn't being used by the frontend anyway.

Removing interval from logger calls in counts.py is not a big loss since we
now include the frequency (which is typically also the interval) in
CountStat.property.
2017-04-14 11:41:07 -07:00
Rishi Gupta b555191ee7 analytics: Change subgroup and labels lists to dictionary in views.py.
A dict is more semantically appropriate than two lists with an assertion
that they be the same length.
2017-04-13 12:37:47 -07:00
Rishi Gupta f3fc9721f4 analytics: Match client names in populate_analytics_db to populate_db.
Originally, all the client names in populate_analytics_db started with
underscores to make it easy to selectively delete and regenerate them when
re-running populate_analytics_db.

We eventually want to merge populate_analytics_db into populate_db though,
in which case it makes more sense for them to share client names, and not
worry about the case where we run (or re-run) populate_analytics_db
independently of populate_db.
2017-04-12 11:45:15 -07:00
Rishi Gupta 9f60dd8387 analytics: Send zeros for data.user.bot in Messages Sent Over Time.
It will simplify the logic needed to process the "Sent by Me" view in
Messages Sent Over Time in stats.js.

Also, we gzip the data sent from our server, so there is little additional
network usage by doing this.
2017-03-25 14:18:23 -07:00
Tim Abbott 20a7609018 analytics: Rename message count types to use standard Zulip casing. 2017-03-21 00:09:54 -07:00
hollywoodno dd067c761a analytics: Separate private messages from group private messages.
This makes it possible for our graphs to show the group private
message counts as separate from 1:1 private messages.

Fixes #4102.
2017-03-20 11:46:29 -07:00
Tim Abbott 0c0e5397c4 analytics: Fix nondeterministic ordering of labels. 2017-03-20 11:39:08 -07:00
Umair Khan 4442703011 jinja2: No need for custom render_to_response.
Django 1.10 has changed the implementation of this function to
match our custom implementation; in addition to this, we prefer
render().

Fixes #1914 via #4093.
2017-03-17 13:57:34 -07:00
Umair Khan 6511f929cc analytics: Change render_to_response to render.
Related to #4093
2017-03-17 13:52:59 -07:00
Rishi Gupta ebebd04587 analytics: Fix ValueErrors affecting test coverage.
Pathways that only catch internal code errors should use AssertionError so
that they are not included when computing test coverage.
2017-03-14 16:59:54 -07:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 2b2be8120f Change datetime.now(tz=X) to timezone.now().
datetime.now with a timezone set is equivalent to timezone.now() if it's
never being printed out, but the latter is cleaner and more idiomatic.
2017-03-01 22:54:28 -08:00
Rishi Gupta d6ce017a58 analytics: Minor cleanup of views.py. 2017-02-11 14:51:01 -08:00
Rishi Gupta 480fc0874b analytics: Break ties deterministically in sort_client_labels. 2017-02-11 14:51:01 -08:00
Tim Abbott f944ac8902 mypy: Fix incorrect annotation for by_used_time. 2017-02-10 23:53:44 -08:00
Rishi Gupta 19d1fc6223 stats: Pass user data to the frontend for messages sent over time. 2017-02-10 14:41:18 -08:00
Rishi Gupta 68a7f91022 stats: Add a fixed display order to summary charts.
API: Adds a "display_order" to the response, which is a suggested order of
importance for the clients or recipient types respectively.

frontend: Changes messages_sent_by_{client,recipient_type} to use a fixed
order for any given user.
2017-02-10 14:41:18 -08:00
Rishi Gupta cf3ae2eafe stats: Turn messages_sent_by_client into a bar chart.
Also includes a number of changes to messages_sent_by_recipient_type that
were convenient to do at the same time, since the two charts share a lot of
code.
2017-02-10 14:41:18 -08:00
Rishi Gupta ce89c64f43 stats.js: Move name_map computation to the backend. 2017-02-10 14:41:18 -08:00
Rishi Gupta a1b1ffe1e4 analytics: Base default views end_time on FillState, not current time. 2017-02-10 14:41:07 -08:00