Commit Graph

16970 Commits

Author SHA1 Message Date
Rishi Gupta 49bd330304 analytics: Add class DependentCountStat and stat realm_active_humans::day. 2017-04-14 11:41:07 -07:00
Rishi Gupta 62de1cf898 test_counts: Modernize TestProcessCountStat tests. 2017-04-14 11:41:07 -07:00
Rishi Gupta 1e8d2b984d counts.py: Rename DataCollector-level operations to be more generic.
We're about to use these for DependentCountStats that will run SQL queries
on the analytics tables instead of the zerver tables.
2017-04-14 11:41:07 -07:00
Rishi Gupta 47cf1d15ba counts.py: Move performance logging call out of pull_functions.
Makes it less likely someone will write a pull function in the future and
forget.
2017-04-14 11:41:07 -07:00
Rishi Gupta 6dff22cbaf counts.py: Change check for LoggingCountStat to use isinstance.
I think this is more pythonic?

We could also get rid of LoggingCountStats altogether, since it's now just a
special case of CountStat (is_logging == data_collector.pull_function is None).
But I think it's nice to keep the distinction since they behave so differently.
2017-04-14 11:41:07 -07:00
Rishi Gupta b45185562a counts.py: Fix out of date comments. 2017-04-14 11:41:07 -07:00
Rishi Gupta ac2cc9e2da counts.py: Reorganize file into logical sections.
No changes to code or behavior.
2017-04-14 11:41:07 -07:00
Rishi Gupta 50868b98a9 counts.py: Change pull_function to take a property instead of a full stat.
Removes the circular dependency of CountStat containing a DataCollector, and
DataCollector containing a function that takes a CountStat as an argument.
2017-04-14 11:41:07 -07:00
Rishi Gupta eadfc743c8 counts.py: Remove CustomPullCountStat. 2017-04-14 11:41:07 -07:00
Rishi Gupta 118b44d4f0 counts.py: Change DataCollector to take a pull_function argument.
This will allow us to appropriately generalize CountStat to include
LoggingCountStat and CustomPullCountStat. It'll also make life easier when
we introduce DependentCountStat.
2017-04-14 11:41:07 -07:00
Rishi Gupta f9e56ad25d counts.py: Move DataCollector declarations into CountStat declarations.
The previous zerver_* names were unwieldy and not very readable. This also
puts more of the useful information in one place; in particular, makes it
easier to skim a CountStat declaration and see if we're collecting it at a
user/stream granularity or a realm granularity.
2017-04-14 11:41:07 -07:00
Rishi Gupta c20e79ab1f counts.py: Rename DataCollector.analytics_table to output_table. 2017-04-14 11:41:07 -07:00
Rishi Gupta 6369d23633 counts.py: Rename ZerverCountQuery to DataCollector.
Not the final form of DataCollector, but the name change causes a big diff
so separating it out.
2017-04-14 11:41:07 -07:00
Rishi Gupta b3991e2557 counts.py: Move CountStat.group_by into ZerverCountQuery.
Part of a larger refactoring to reduce cyclic dependencies between CountStat
and DataCollector (coming soon).
2017-04-14 11:41:07 -07:00
Rishi Gupta 341e1b54fc counts.py: Remove zerver_table from ZerverCountQuery.
Was only needed for filter_args, which are now gone.
2017-04-14 11:41:07 -07:00
Rishi Gupta 661de6bf25 counts.py: Remove filter_args argument from CountStat definition.
It turned out to not be that useful once we added subgroup. The previous
design of the CountStat object also assumed more reuseability of the *_query
strings than what ended up happening.

The filter_args also had some carrying costs:

* It's hard to be confident that filter_args other than the ones explicitly
  in our tests would have had expected behavior.
* The filter_args/join_args system is the most complex part of the CountStat
  object, and makes understanding the *_query strings unnecessarily
  difficult for a new contributor.
2017-04-14 11:41:07 -07:00
Rishi Gupta 4dfadba244 counts.py: Hardcode is_active=true in count_user_by_realm_query.
A step towards removing filter_args from the CountStat object.
2017-04-14 11:41:07 -07:00
Rishi Gupta 6bb97db136 analytics: Add active_users_audit:is_bot:day. 2017-04-14 11:41:07 -07:00
Rishi Gupta cc75d83b74 counts.py: Reorder count_stats_ to put similar stats together. 2017-04-14 11:41:07 -07:00
Rishi Gupta b396be4c98 models: Add index on RealmAuditLog.event_time.
Useful for the upcoming check_realmauditlog_by_user_query, if nothing else.

But I suspect it will indeed get use; looking for events around or within a
certain time is pretty natural for an audit log.

The main argument against I would say is that this should actually be a
joint index with something else. I'm not sure what that something else
should be, so just optimizing for what I think
check_realmauditlog_by_user_query will need for now.
2017-04-14 11:41:07 -07:00
Rishi Gupta 3d514c3e8d analytics: Add a default for the value column in assertTableState.
A default value of 1 is reasonable in this framework, especially for testing
things like LoggingCountStats.
2017-04-14 11:41:07 -07:00
Rishi Gupta ce376261ca models: Add comment explaining RealmAuditLog.backfilled. 2017-04-14 11:41:07 -07:00
Rishi Gupta 2f74ccabf9 analytics: Add 15day_actives CountStat. 2017-04-14 11:41:07 -07:00
Rishi Gupta 9b661ca91f analytics: Replace CountStat.is_gauge with interval.
Groundwork for allowing stats like "Monthly Active Users".

CountStat.interval is no longer as clean a value as before, so removed it
from views.get_chart_data. It wasn't being used by the frontend anyway.

Removing interval from logger calls in counts.py is not a big loss since we
now include the frequency (which is typically also the interval) in
CountStat.property.
2017-04-14 11:41:07 -07:00
Rishi Gupta d6c5c672d3 analytics: Add minutes_active CountStat. 2017-04-14 11:41:07 -07:00
Joshua Pan 6be482b0e7 Link /help/ documentation from in-app documentation.
Fixes: #4455.
2017-04-14 11:35:40 -07:00
Brock Whittaker fe8f63c389 message view: Fix rare cases where click-to-reply was not handled.
The comment explains this change is fairly good detail.  This change
is designed to fix complaints that sometimes clicking on a message to
reply doesn't work, which we can reproduce as being caused by clicks
that happen simultaneous with a few pixels of mouse motion at the same
time as the click.

Comment and commit message rewritten by tabbott for clarity.
2017-04-14 11:25:15 -07:00
hackerkid cd5334c827 Show local time of user in user_popover. 2017-04-14 10:38:29 -07:00
hackerkid 15cdd23525 Add option for setting timezone in user display settings. 2017-04-14 10:38:21 -07:00
hackerkid bf3b2ac673 Include timezone in user_dict fields.
Tweaked by tabbott to avoid adding timezone to bot dicts, since bots
don't need a timezone.
2017-04-14 10:33:55 -07:00
hackerkid 2a0a84b229 Add moment.js third-party module via npm.
This is needed for timezone manipulations.  We may be able to replace
XDate with it in the future as well.
2017-04-14 10:22:06 -07:00
derAnfaenger ecf9a50610 Clarify UI description of message retention. 2017-04-14 10:13:34 -07:00
Tim Abbott a417fd3c0b MessageDictTest: Fix nondeterministic query counts.
This fixes an issue with a nondeterministic number of database queries
being used in fetching bulk messages from the database.  The source of
the problem was that we were fetching _all_ messages, not just the 600
that had been created by the test, and thus if the set of streams
present in messages in the test fixtures (which is random) changes,
the number of streams used (and thus number of queries) would change.
2017-04-14 09:55:27 -07:00
Tim Abbott 622b7f00f7 test_runner: Fix unnecessary sleep(1) in database teardown.
Apparently, Django's _destroy_test_db has a mostly unnecessary
sleep(1) before dropping the database, which obviously wastes a bunch
of time in the single-test runtime of their database teardown logic.

We work around this by monkey-patching that function to not do the sleep.
2017-04-14 09:30:47 -07:00
Umair Khan a507a47778 testing: Use zulip_test_template for backend.
Instead of zulip_test, use zulip_test_template for backend DB. This
makes sure that the DB used by backend tests is different from the
DB, which will be zulip_test, used by Casper tests.
2017-04-14 10:23:31 +05:00
Umair Khan bf713bcdfe testing: Don't create DB twice for backend.
In backend tests, only call generate-fixtures when --generate-fixtures
is explicitly passed or is_template_database_current() returns False.

We don't need to flush cache for backend tests because we bounce the key
prefix used to create cache keys before running every test
2017-04-14 10:23:16 +05:00
Steve Howell e23b660ccc compose: Extract set_focus().
Most of the code here was in show_box(), but we also pull in the
call to get_focus_area() from compose.start().
2017-04-13 17:13:19 -07:00
Steve Howell 0841320966 compose: Extract maybe_scroll_up_selected_message().
The extraction here is straightforward, but where we put the
caller is a slightly subtle change.  Instead of continuing to
invoke this code at the end of show_box(), we instead call it
at the beginning of complete_starting_tasks().  This change is
valid, because show_box() and complete_starting_tasks() are only
ever called from compose.start().
2017-04-13 17:13:19 -07:00
Steve Howell aa5409cf3a compose: Extract complete_starting_tasks(). 2017-04-13 17:13:19 -07:00
Steve Howell e73b56c7b5 compose: Extract expand_compose_box(). 2017-04-13 17:13:19 -07:00
Steve Howell 79ff32a01e compose: Extract autosize_message_content(). 2017-04-13 17:13:19 -07:00
Umair Khan 519dcdb750 api_dev_fetch_api_key: Improve invalid email message.
Show a user friendly message to the user if email is invalid.
Currently we show a generic message:
"Your username or password is incorrect."
2017-04-13 12:48:13 -07:00
Umair Khan 80b019629c remote_user_sso: Improve invalid email message.
Show a user friendly message to the user if email is invalid.
Currently we show a generic message:
"Your username or password is incorrect."
2017-04-13 12:48:13 -07:00
Umair Khan 93aa478efb api_fetch_api_key: Improve invalid email message.
Show a user friendly message to the user if email is invalid.
Currently we show a generic message:
"Your username or password is incorrect."

The only backend which can accept a non-email username is LDAP.
So we check if it is enabled before showing the custom message.
2017-04-13 12:48:13 -07:00
Umair Khan ad2114a7a8 validator.py: Create a validator for login email.
This validator raises JsonableError exception.

Fixes: #2748
2017-04-13 12:48:13 -07:00
Rishi Gupta 6e425814bf analytics: Add a few tests for fixtures.py.
The code in fixtures.py is only called from populate_analytics_db, and is
only used for generating pretty fixture data for manual testing. This commit
adds tests for a few things that were easy to add tests for, and provides
some minimal coverage of the file, but is not meant to be comprehensive.
2017-04-13 12:37:47 -07:00
Rishi Gupta b555191ee7 analytics: Change subgroup and labels lists to dictionary in views.py.
A dict is more semantically appropriate than two lists with an assertion
that they be the same length.
2017-04-13 12:37:47 -07:00
Steve Howell f06bd41586 Fix PM list sort ordering during scrollback situations.
Before this fix, if you scrolled back in your PM history for a
person that you've had recent conversations with, then we would
backdate the record of their most recent conversation, and this
would make the sort ordering under the "Private messages"
section incorrect.

This commit fixes this error by re-writing the function
message_store.insert_recent_private_message() to check any
prior timestamps for that user.  It also optimizes the function
a bit to short-circuit in O(1) time for cases where a recipient
already has a more recent timestamp, by having a Dict keyed
on user_ids_string.
2017-04-13 12:13:20 -07:00
Steve Howell 24461762da Add test_insert_recent_private_message(). 2017-04-13 12:13:20 -07:00
Steve Howell f5550301cf Extract message_store.insert_recent_private_message().
This function is slightly easier to unit test, and it isolates us
from changing message formats.  This removes some extraneous
code that would ensure that message timestamps were >= 0 that
probaby dates back to some really old migrations.
2017-04-13 12:13:20 -07:00