Commit Graph

6002 Commits

Author SHA1 Message Date
Greg Price a116303604 passwords: Express the quality threshold as guesses required.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful.  For configuration and the core
accept/reject logic, it's better to use units that are readily
understood.  Switch to those.

I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses.  And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft".  So, "guesses" it is.

And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting.  So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.

(Thanks to @YJDave for a prototype that the code changes in this
 commit are based on.)
2017-10-08 15:48:44 -07:00
rht fbd24669a4 Cleanup: Remove shebang and executable flag from these files. 2017-10-08 15:38:15 -07:00
Tim Abbott ec080aed6b mypy: Workaround lxml annotations being busted. 2017-10-08 12:38:20 -07:00
Tim Abbott d215ea1e37 actions: Rename all_subs_by_stream to all_subscribers_by_stream.
The previous name sounded a bit too much like they were subcription
objects.
2017-10-08 12:33:53 -07:00
Steve Howell 3e6bfe1b23 Use user_ids, not emails, for bulk stream operations.
We now return user_ids for subscribers to streams in add-stream
events.  This allows us to eliminate the UserLite class for
both bulk adds and bulk removes.  It also simplifies some JS
code that already wanted to use user_ids, not emails.

Fixes #6898
2017-10-08 12:31:12 -07:00
Harshit Bansal 3c434f0d86 notifications: Switch to use `make_links_absolute()` from lxml library.
Instead of using custom regexes for converting relative URLs to
absolute URLs switch to using `make_links_absolute()` function
from lxml library.
2017-10-08 12:15:30 -07:00
Steve Howell 10a30bece1 Rename presence_idle_userids -> presence_idle_user_ids. 2017-10-07 12:16:45 -07:00
Steve Howell fbaef43ac3 Rename bot_owner_userids -> bot_owner_user_ids. 2017-10-07 12:16:45 -07:00
Greg Price aa4104a5af logging: Add option to show the PID in each log message. 2017-10-06 19:21:40 -07:00
Harshit Bansal 7b88e3eea7 tests: Add a test suite for verifying the mobile push notifs content.
This test suite works by using the expected_output and new text_output
fields in the bugdown test cases to verify that each syntax is
correctly translated by this new function.

Some of these translations, like strikethrough, are kinda poor; but
this framework should make it easy to iterate on the formatting.

Fixes: #6720.
2017-10-06 16:47:27 -07:00
Harshit Bansal 5a6584890d push_notifications: Start using `get_mobile_push_content()` function. 2017-10-06 16:47:25 -07:00
Harshit Bansal 28628eeaeb push_notifications: Add `truncate_content()` function.
This function truncates the textual content at correct length.
(It will be updated later to handle corner cases of unicode
combining characters and tags when we start supporting them.)
2017-10-06 16:44:19 -07:00
Harshit Bansal b5a1aacfb3 push_notifications: Add `get_mobile_push_content()` function.
Given the rendered content of a message, this function strips
all the markup replacing emojis with their corresponding unicode
representation.
2017-10-06 16:44:18 -07:00
Tim Abbott 234f7960a1 tests: Add @slow decorator to various methods needing it.
Also, fix the test_message_editing test having way too many cases.

Fixes #1478.
2017-10-06 15:45:04 -07:00
Tim Abbott 0d7617721c test_messages: Clean up some slow tests.
One isn't actually slow; the other was doing something wasteful.
2017-10-06 15:44:17 -07:00
Tim Abbott d5c2508858 tests: Remove unnecessary @slow decorators.
This is part of addressing #1478.
2017-10-06 15:17:34 -07:00
Steve Howell 9202777d7f tests: Provide more useful output in assert_length(). 2017-10-06 14:30:30 -07:00
Steve Howell d2966ae5d4 minor: Fix missing "not" in comment. 2017-10-06 14:08:41 -07:00
Tim Abbott 73e1c26c15 test_events: Add missing nocoverage tag on debugging code. 2017-10-06 13:59:26 -07:00
Steve Howell ad521a0e07 minor: Add debugging statement to test.
This test is flaky for some unknown reason, so we print
out the loop index in case the test fails.
2017-10-06 13:56:37 -07:00
Steve Howell a99574ebf0 tests: Make apply-events tests easier to debug.
It's fairly difficult to debug tests that use
EventsRegisterTest.do_test, and when they fail on
Travis, it's particularly challengning.  Now we make
the main diff less noisy, and we also include
the events that were applied.
2017-10-06 13:56:36 -07:00
derAnfaenger 747c283180 lint: Replace `pk` with `id` where they are equivalent. 2017-10-06 12:52:12 -07:00
Tim Abbott 0c44310009 tests: Don't use caching session engine in tests.
Because we run tests in parallel, the caching session engine can lead
to nondeterministic failures.
2017-10-06 12:22:20 -07:00
Steve Howell a331b4f64d Optimize query_all_subs_by_stream().
Using lightweight objects will speed up adding new users
to realms.

We also sort the query results, which lets us itertools.groupby
to more efficiently build the data structure.

Profiling on a large data set shows about a 25x speedup for this
function, and before the optimization, this function accounts
for most of the time spend in bulk_add_subscriptions.

There's a lot less memory to allocate.  I didn't measure
the memory difference.

When we test-deployed this to chat.zulip.org, we got about a 6x
speedup.
2017-10-06 11:03:44 -07:00
Steve Howell f5ddc40d14 Have get_peer_user_ids_for_stream_change() use user_ids. 2017-10-06 11:03:44 -07:00
Tim Abbott fe6a04cbdb Revert "locale: Fix incorrect locale name for zh-Hans."
This reverts commit ba8dc62132.

As best I can tell, the old configuration was correct for what Django
wanted.  Further testing is required, but this at least brings
.tx/config to match the actual filenames; I think our Chinese
translations have been broken until now.
2017-10-06 10:09:08 -07:00
Tim Abbott 0f2e4a55c0 soft deactivation: Shorten management command name.
This command is really for soft deactivation; there's just an undo
feature.
2017-10-06 08:48:43 -07:00
Tim Abbott ca59b28fc1 import: Fix positional argument parsing. 2017-10-06 08:45:22 -07:00
Tim Abbott bfedb82097 import: Fix argparse usage message breakage. 2017-10-06 08:41:01 -07:00
Tim Abbott 1a9adbbe70 test_signup: Fix test for login page redirect.
I forgot to update this in 43bba27e5d.
2017-10-05 23:36:33 -07:00
Tim Abbott 313083484a test_i18n: Remove zh-hans tests for now. 2017-10-05 23:33:12 -07:00
Tim Abbott 43bba27e5d auth: Fix redirect loop in development environment.
Since the REALMS_HAVE_SUBDOMAINS migration in development, we've had
scattered reports of users who found trying to open 127.0.0.1:9991
resulting in a redirect loop between zulipdev.com:9991,
zulipdev.com:9991/devlogin, and zulipdev.com:9991/devlogin/, and back
to zulipdev.com:9991.

We fix this temporarily through a small cleanup, which is to have that
last step in the loop send the user to the subdomain where they're
actually logged in, zulip.zulipdev.com:9991.

There's more to be done before this system will make sense, though.
2017-10-05 23:21:29 -07:00
Tim Abbott 46485322eb middleware: Remove logic for redirecting to zulipdev.com domains.
We originally wrote this because when testing subdomains, you wanted
to be sure you were actually testing subdomains.  Now that subdomains
is the default, doesn't seem to actually be a good reason why we
should need this.
2017-10-05 23:21:02 -07:00
Umair Khan 6fbf41bdbc queue: Add test for signups retry logic. 2017-10-05 23:14:19 -07:00
Umair Khan 435fe40199 worker: Retry signups queue event on 400. 2017-10-05 23:14:19 -07:00
Umair Khan 19e2551e82 mypy: Change type to Dict for SignupWorker. 2017-10-05 23:14:19 -07:00
Tim Abbott f11832536c event_queue: Fix a now-incorrect mypy annotation.
It's not clear tagging things like this as an Iterable is a good idea,
when it's in fact definitely a list.
2017-10-05 23:11:16 -07:00
Umair Khan 326a6f6b4f i18next: Don't allow empty string as valid translation.
Previously we used to mark a key as unstranlated if its value was equal
to it in translations.json. This had an issue because it didn't allow
otherwise valid cases where key was equal to the value.

This commit solves the problem by disallowing an empty string as a valid
translation and then using the empty string as the value for all the
unstranslated keys.

Fixes #5261
2017-10-05 22:44:34 -07:00
Umair Khan b464fd7643 makemessages: Change type of translation_strings to List.
In the code this variable is only used as a list, however, we
create it as dictionary which uses more memory.
2017-10-05 22:44:34 -07:00
Tim Abbott fbdeb63bc9 tornado: Add debug logging for time consumed processing an event.
I've occasionally found this logging helpful.
2017-10-05 22:08:29 -07:00
Tim Abbott 40c59f2878 middleware: Fix losing sub-URL when pushing to zulipdev.com.
Previously, this would always send one to homepage, making visiting
the /help/ documentation in the development environment using the
localhost URL unpleasant.

While this fixes the proximal bug, it's not clear to me that we need
this redirect logic at all, so I'm going to try removing it soon.
2017-10-05 16:36:34 -07:00
Alena Volkova 21bc0e845c urls: Delete the json/tutorial_send_message endpoint.
This endpoint is part of the old tutorial, which we've removed, and
has some security downsides as well.

This includes a minor refactoring of the tests.
2017-10-05 15:41:57 -07:00
Tim Abbott e1eb91343b zephyr: Update path to process_cache for repository split.
This should have been done long ago, but better late than never.
2017-10-05 12:35:20 -07:00
Rishi Gupta 0596c4a810 analytics: Enforce various datetime arguments are in UTC.
Sort of a hacky hammer, but
* The original design of the analytics system mistakenly attempted to play
  nicely with non-UTC datetimes.
* Timezone errors are really hard to find and debug, and don't jump out that
  easily when reading code.

I don't know of any outstanding errors, but putting a few "assert this
timezone is in UTC" around will hopefully reduce the chance that there are
any current or future timezone errors.

Note that none of these functions are called outside of the analytics code
(and tests). This commit also doesn't change any current behavior, assuming
a database where all datetimes have been being stored in UTC.
2017-10-05 11:22:06 -07:00
Rishi Gupta 0c2b4d22a7 analytics: Convert datetimes coming from the API into UTC.
Previously, entering a non-UTC end time for a daily stat would give you
incorrect results. This is because:
* All daily stats are collected at and have end_times in the database in
  midnight UTC.
* For daily stats, time_range returns a list of datetimes at midnight in the
  timezone of its end argument. These datetimes are the only ones we look
  for when looking for rows corresponding to the stat in the database.
* Previously, we passed on the end argument from the API to time_range,
  without modification.
2017-10-05 11:22:06 -07:00
Rishi Gupta 70f6c47edc analytics: Extract verify_utc into its own function.
No functional changes, other than making the error message more generic.
2017-10-05 11:22:06 -07:00
Steve Howell d6e21b5ca9 Collect sender_ids (by topic) in `unread_msgs`.
This will allow the mobile app to say "A, B, and C are
talking" in the topic views.
2017-10-05 10:37:15 -07:00
Tim Abbott 636c9429f6 email_log: Don't require full coverage for error handling path. 2017-10-05 10:34:56 -07:00
Steve Howell e56084fcf7 Simplify how we apply events for unread messages.
The logic to apply events to page_params['unread_msgs'] was
complicated due to the aggregated data structures that we pass
down to the client.

Now we defer the aggregation logic until after we apply the
events.  This leads to some simplifications in that codepath,
as well as some performance enhancements.

The intermediate data structure has sets and dictionaries that
generally are keyed by message_id, so most message-related
updates are O(1) in nature.

Also, by waiting to compute the counts until the end, it's a
bit less messy to try to keep track of increments/decrements.
Instead, we just update the dictionaries and sets during the
event-apply phase.

This change also fixes some corner cases:

    * We now respect mutes when updating counts.
    * For message updates, instead of bluntly updating
      the whole topic bucket, we update individual
      message ids.

Unfortunately, this change doesn't seem to address the pesky
test that fails sporadically on Travis, related to mention
updates.  It will change the symptom, slightly, though.
2017-10-05 09:42:20 -07:00
Steve Howell c567f105c9 Have topic_is_muted take a stream_id.
This function doesn't need a full Stream object to detect
whether a stream is muted, so we can save future callers
from doing unnecessary DB fetches.
2017-10-05 09:32:16 -07:00