Commit Graph

468 Commits

Author SHA1 Message Date
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Vishnu KS ec955f8f78 support: Show confirmation links in search.
Fixes #13060 #12784
2019-10-21 16:56:50 -07:00
Vishnu KS 139ebf387b support: Pass various realm functions as template context.
We currently have code to calculate the value of realm_icon_url,
admin_emails and default_discount in two diffrent places. With
the addition of showing confirmation links it would become three.
The easiest way to deduplicate the code and make the view cleaner
is by doing the calculations in template. Alternatively one can
write a function that takes users, realms and confirmations as
arguments and sets the value of realm_icon_url, admin_emails and
default_discount appropriately in realm object according to the
type of the confirmation. But that seems more messy than passing
the functions directly to template approach.
2019-10-21 16:52:46 -07:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Tim Abbott a352f2e10d test_counts: Remove custom user creation code.
Like the last commit, this avoids us needing to update this random
test class in the analytics subsystem when we adjust the UserProfile
model.
2019-09-19 14:31:58 -07:00
Tim Abbott aae610f65c analytics: Fix creation of shylock user.
Previously, the shylock user was created using a bad duplicate of our
standard create_user code path; we fix this by just calling into that
code path.
2019-09-19 14:16:38 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 490ce993f4 analytics: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Wyatt Hoodes 97c10b35c2 analytics/views: Remove redundant check of `if end is None`. 2019-07-31 12:15:57 -07:00
Wyatt Hoodes fa227d79af typing: Fix options typing in management/commands.
We simply state that certain options are `Optional`.

The following files are affected:

add_users_to_mailing_list
send_to_email_mirror
fill_memcached_caches
client_activity
2019-07-31 12:13:56 -07:00
Mateusz Mandera b321dcd50b test_views: Prepare for moving system bots to zulipinternal. 2019-07-24 16:26:10 -07:00
Rishi Gupta 9fc791926b support: Use delivery_email instead of email.
Structurally, these pages want email addresses one can actually send
emails to.
2019-07-21 14:47:15 -07:00
Tim Abbott 4eb9d67b70 models: Extract get_human_admin_users function.
This function is an alternative to get_admin_users that we use in all
places where we explicitly want only human administrative users (not
administrative bots).  The following commits will rename
get_admin_users for better clarity.
2019-06-20 14:32:30 -07:00
Mohit Gupta db3d81613b decorator: Refactor @require_non_guest_human_user decorator.
Rename @require_non_guest_human_user to @require_member_or_admin.
This is a refactor commit prior to introduction of Administrator
Bots.
2019-06-18 17:11:58 -07:00
Vishnu Ks 14e582fb59 support: Add functionality to copy admin emails.
Also renamed a bunch of functions in test_views for better
readability.
2019-06-14 10:19:50 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Tim Abbott 1d44fd724b audit log: Log which server admin deactivated a realm too. 2019-05-08 15:09:48 -07:00
Rishi Gupta 0218da64f1 support: Update success message for plan type change. 2019-05-08 15:09:48 -07:00
Rishi Gupta 98da11c558 support: Rename deactive to deactivated. 2019-05-08 15:09:48 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Vishnu Ks 6c58603eaf support: Add support for scrubbing realm. 2019-05-06 20:12:54 -07:00
Vishnu Ks f6203f068b support: Add support for activating and deactivating realm. 2019-05-06 20:12:48 -07:00
Vishnu Ks 123bcea518 management: Don't use sys.exit(1).
Using sys.exit in a management command makes it impossible
to unit test the code in question.  The correct approach to do the same
thing in Django management commands is to raise CommandError.

Followup of b570c0dafa
2019-05-03 14:20:39 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Vishnu Ks 8eeb8280b4 activity: Create interface for doing support operations.
This should grow into a tool that makes it much easier to do common
organization management tasks without using a manage.py shell.
2019-03-11 12:01:11 -07:00
Vishnu Ks 42de9a0c71 analytics: Extract out the function for getting plan name. 2019-03-11 12:01:11 -07:00
Harshit Bansal 0e401c4f18 minor: Fix an artifact of delivery email migration in `populate_analytics_db`. 2019-03-05 13:52:59 -08:00
Rishi Gupta 9962377018 analytics: Fix midnight-related bug in test.
Previously, this would flake if the day changed between
user2 = do_create_user('email2', 'password', ...) and
do_deactivate_user(user2).
2019-02-28 14:48:30 -08:00
Tim Abbott 216d2ec1bf production: Add optional support for submitting usage statistics.
See documentation for details.
2019-02-26 17:35:10 -08:00
Tim Abbott c41bfcb9e0 Revert "activity: Change definition of active site."
This reverts commit 9f9b7cb991.

This commit made the page not perform well enough to load.
2019-02-13 14:52:13 -08:00
bartek 9f9b7cb991 activity: Change definition of active site.
Signed-off-by: bartek <bartek.jachowicz@gmail.com>

Edits by Rishi Gupta <rishig@zulipchat.com>

Fixes: #10432
2019-02-12 16:36:44 -08:00
Wyatt Hoodes bb6a75c3dc populate_analytics_db.py: Subscribe user for exporting.
AssertionErrors were raised when attempting to run manual comparison
tests to ensure correctness when exporting the analytics realm using
export_from_config.  This was caused by this populate_analytics_db
stream being created without any subscribers, which violates an
invariant.

We fix this by simply subscribing the 'shylock' user to that stream.
2019-02-04 10:59:24 -08:00
Anders Kaseorg c2cb804ebe analytics/views.py: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:43:58 -08:00
Anders Kaseorg f5197518a9 analytics/zilencer/zproject: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:31:45 -08:00
Tim Abbott 13b9e90a40 analytics: Fix importing zilencer.
In ebdd55814c, we added zilencer imports
without using the proper mocking procedure for when zilencer is not enabled.

This whole setup is a mess and probably we should enable zilencer
unconditionally in a future version.
2019-02-02 17:18:57 -08:00
Anders Kaseorg 4bd28f7ae6 migrations: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:01:04 -08:00
Tim Abbott 96f096f38e analytics: Add section for remote Zulip servers. 2019-02-02 16:55:12 -08:00
Tim Abbott ebdd55814c analytics: Add support for graphing remote analytics data.
Combined with recent work on submitting data, one can now view the
high-level usage numbers for chat.zulip.org on zulipchat.com's
dashboards.
2019-02-02 16:55:12 -08:00
Rishi Gupta 85f7ac8172 analytics: Remove Anomaly model. 2019-02-01 18:48:18 -08:00
ss62171 5649b954ef stream_stats: Add a column representing type of stream.
This adds a column which represents whether a stream is public or
private.

Fixes #11374.
2019-01-31 15:04:45 -08:00
ss62171 5fd9748e13 stream_stats: List number of private and public streams for each realm.
Previously, we only displayed public streams and didn't list the
number of them.  Now, we just list all streams.
2019-01-31 15:04:09 -08:00
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 7ddcbd0d3a analytics: Set delivery_email in hacky test user creation code. 2018-12-06 15:33:28 -08:00
Tim Abbott c679920c01 python: Fix unnecessary uses of str_utils library. 2018-11-27 11:44:09 -08:00
Rishi Gupta 91b02373dc activity: Rename standard free to open source.
"standard free" looks too similar to "standard". Also it makes it hard to
Ctr+f for "standard".
2018-11-16 21:03:37 -08:00
Vishnu Ks 2e04cdbe5e billing: Show estimated subscription revenue on /activity.
[Substantial edits by Rishi Gupta]
2018-11-16 13:30:16 -08:00
Vishnu Ks 9d398ee335 activity: Move the total row from last to first. 2018-11-16 09:22:32 -08:00
Steve Howell c0fd8660d2 subject -> topic: Fix analytics test (minor). 2018-11-14 23:24:06 -08:00
Yashashvi Dave 02a5849d4c statistics: Guest user can't access realm statistics.
Don't allow guest user to access realm statistics from
UI or at API level.

Fixes part of #10749.
2018-11-02 11:43:09 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Tim Abbott 0deeffff6d populate_analytics_db: Make shylock an organization admin.
This means the analytics realm satisfies the usual Zulip invariant
that every realm has at least one organization admin.
2018-08-31 15:26:15 -07:00
Vishnu Ks 6fcb095e70 activity: Show plan_type in activity page. 2018-08-21 18:15:14 -07:00
Roman Godov 34ae3dfd44 models: Delete unused Subscription.notifications field.
This deletes the unused Subscription.notifications field and removes
it from some testing and analytics code (which should not have been
using it in the first place).

Fixes #10042.
2018-07-26 15:54:57 -07:00
Anders Kaseorg 8d52f0e0c0 analytics/management/commands/check_analytics_state.py: Avoid shelling out for mv.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Vishnu Ks 201b99a6f8 models: Add USER_REACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d0b89cbb44 models: Add USER_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks ce3fffdbb2 models: Add USER_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 2c8effe9fe models: Add USER_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 4d1a68430a analytics: Remove unused RealmAuditLog import. 2018-07-10 15:42:26 +05:30
Tim Abbott b9fa7d7b6d populate_analytics_db: Flush memcached after deleting analytics realm.
This fixes a subtle bug where if you reran populate_analytics_db
directly, we'd end up in a weird state where memcached fetched the
"old" pre-flush UserProfile object for shylock when loading /stats,
which ultimately would result in /stats appearing totally broken.
2018-07-09 15:13:59 +05:30
Eeshan Garg 0a43e5e257 Replace all user-facing references to "invite-only" with "private".
Fixes #9611.
2018-06-12 13:37:45 -04:00
Nikhil Kumar Mishra fa9d79e203 stats: Add 1 day actives and total users to number of users chart. 2018-05-20 10:56:16 -07:00
Rishi Gupta 66a589c7a7 stats: Extend get_chart_data to support charts with multiple CountStats. 2018-05-20 10:56:16 -07:00
Rishi Gupta 08bf0a66b8 stats: Refactor the get_time_series_by_subgroup calls in get_chart_data.
This code is going to end up pretty complex -- each stat has multiple levels
of aggregation (UserCount, RealmCount, InstallationCount), and refinement
(subgroups), and soon we'll have charts that take data from multiple stats
as input.

Not sure what the best way to present it is, but hopefully this simplifies
it a bit.
2018-05-20 10:56:16 -07:00
Nikhil Kumar Mishra 26decb4c48 stats: Add 1day_actives::day CountStat to analytics tables. 2018-05-20 10:56:16 -07:00
Rishi Gupta 1af7fc7344 stats: Add /stats/installation. 2018-05-18 15:12:36 -07:00
Rishi Gupta 2fe3fba6ce stats: Rename data.realm to data.everyone.
We use "Everyone" for the button labels already.

Soon we'll support "Everyone" meaning either the installation or the realm,
depending on the URL route used to access the stats.
2018-05-18 15:12:36 -07:00
Rishi Gupta af758755bd stats: Rename target_realm_name to target_name in stats.html. 2018-05-18 15:12:36 -07:00
Rishi Gupta e099959a41 stats: Move API route computation from frontend to backend.
Will make it easier to extend to additional routes.
No changes in behavior.
2018-05-18 15:12:36 -07:00
Tim Abbott f0ef335412 models: Remove unused ModelReprMixin class.
It appeared to be used as a base class in various Django migrations,
but because it didn't define any model fields, it wasn't actually.
2018-05-15 19:11:22 -07:00
Yago González 184bd8304e i18n: Tag missing strings for translation. 2018-05-12 16:44:56 -07:00
Aditya Bansal 5adf983c3c analytics: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Shubham Dhama 03a2a9c792 activity: Add realm stats link to "realm activtiy table". 2018-04-18 11:07:00 -07:00
Shubham Dhama b26c38bc47 analytics: Make stats of all realms accessible to server admins.
In this commit:
Two new URLs are added, to make all realms accessible for server
admins. One is for the stats page itself and another for getting
chart data i.e. chart data API requests.
For the above two new URLs corresponding two view functions are
added.
2018-04-18 11:06:50 -07:00
neiljp (Neil Pilgrim) 8b697b4093 mypy: Annotate stream_data in populate_analytics_db.py handle function. 2018-03-25 08:59:08 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
Archana BS b5a860b234 analytics: Populate messages_in_stream:is_bot:day in dev. 2018-03-04 13:17:00 -08:00
Tim Abbott 6bbce451bc analytics: Fix a minor mypy error. 2018-02-12 11:57:47 -08:00
Aman Jain a0b58b1560 activity: Add a link to copy list of realm admins.
Tweaked by tabbott to do the database queries properly.

This should help user to copy realm admin emails in a go.
Fixes: #7885
2018-02-12 08:55:45 -08:00
Rishi Gupta 1d581a9c6e nagios: Add nagios check for analytics state.
This should help us detect issues where the analytics cron jobs aren't
running properly.

The cron/nagios part of the implementation done by tabbott.
2018-02-09 16:36:05 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
rht 8106a25e61 django-2.0: Add on_delete on ForeignKeys.
In Django 2.0, one must specify the on_delete behavior for all
ForeignKeys explicitly.
2018-01-30 10:53:54 -08:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price ebcf0b4876 logging: Stop having `create_logger` force loglevels to INFO.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels.  Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.

The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
2017-12-12 17:17:07 -08:00
Greg Price 20b2c11830 activity: Show the time the data is from.
I've wanted this when looking at a tab from the day before.
Also provides the date and time in UTC, which is handy for
interpreting some of the data.

Pretty sure this is not the world's cleanest way to do this in the
front-end code.  It'll do for now.
2017-12-12 15:30:03 -08:00
Greg Price bdea0960de activity: Supply a missing field.
In prod this has no effect, but in dev we render a warning here,
which makes the table not look right.  Fix it.
2017-12-12 15:30:03 -08:00
Greg Price d36b1cd2d7 activity: Correct description of message-history figures.
This explains why the first number was usually the smallest!
2017-12-12 15:30:03 -08:00
Rishi Gupta fbd8dde1f8 invitations: Add LoggingCountStat to keep track of sent invitations. 2017-12-06 20:35:50 -08:00
Greg Price 34ee019b20 activity: Consistently define "user" as non-deactivated, non-bot. 2017-11-30 20:43:46 -08:00
Greg Price 0ebfc2fb5c activity: Highlight recently-created realms. 2017-11-30 20:43:46 -08:00
Greg Price 8de34c93dd activity: Show each realm's creation date.
I'd rather have something rich and fancy like a sparkline of
activity... but this is a lot quicker to implement.
2017-11-30 20:43:46 -08:00
Greg Price 2c9b698cb6 activity: Replace "at-risk users" with WAU.
Substantively, this makes the table more readable by grouping users
into expanding sets by level of activity: active in last day, active
in last week, have an account at all.  The class "active in last week",
as opposed to "active in last week but not in last day", makes more
natural comparisons both between realms and for one realm through time,
and it's less sensitive to the details of our definitions.

This also makes the terminology more standard.  We already made that
change in the display, in the previous commit; as we go through the
logic here, we adjust the terminology in the code too.
2017-11-30 20:43:46 -08:00
rht 01885cdedc analytics: Use Python 3 syntax for typing (final). 2017-11-22 12:16:59 -08:00
rht 6c286b5eb6 analytics: Use Python 3 syntax for typing (part 2). 2017-11-22 12:16:58 -08:00
Tim Abbott a0cfe45150 analytics: Wrap some longer lines. 2017-11-17 13:19:48 -08:00